We continue to perform mailbox level cleanups to address the following issues, which we have identified proactively from system logs:
We recommend that those users who experience intermittent errors connecting via POP or IMAP utilize Webmail instead. While we are working to resolve all symptoms, our highest priority is ensuring all affected users have access to their mailbox.
Our initial pass of fixes for mailboxes that we detected as having login/access issues was successful. We will run a pass for newly discovered mailboxes that our logs have indicated are having issues. Once complete, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:
As we continue to progress through our mailbox cleanup efforts, some users may see slowness within Webmail. Some mailboxes may require additional attention to resolve some of the issues experienced, and our team is actively working through those.
We are continuing to focus on maintaining access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. We have begun running a fix to address the issues for users that have not yet been able to successfully access their mailbox. During this time, some users may see some slowness within Webmail. Once completed, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:
Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.
Over the weekend we have worked through a system-wide cleanup to address the after-effects of the high system load experienced last week. The load was the result of an unknown bug that impacted an effort to proactively double system capacity.
Our priority today is to maintain good access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. In an abundance of caution, we are limiting the number of simultaneous connections to a single mailbox to 2 POP connections and 4 IMAP connections. Mail programs that exceed the limit will usually retry and users are unlikely to see errors due to connection limits.
We are also rate limiting email delivery for mailboxes that receive hundreds or thousands of email messages per minute; that email will safely queue for delivery over a longer period of time. If a user normally receives less email than this, they should see normal delivery speeds.
After peak load hours today, usually finished before 2pm Eastern, we will consider a bulk mailbox level cleanup to address several issues we have identified proactively from system logs. These issues include:
Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those. We apologize for these extended issues with the email platform and will post here as more information becomes available.
Our efforts this weekend have been focused on system wide cleanup to address the consequences of high system load after expanding capacity. These efforts are nearing completion. We will soon begin a more detailed mailbox-level cleanup effort to address some issues that we have seen among some users. While those updates happen there may be very brief periods where accessing email is slow or gives an error; trying again a few minutes later should be successful.
The mailbox level cleanup will address, in an automated fashion, the following issues for users we have been able to identify proactively:
Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those. We apologize for these extended issues with the email platform and will post here as more information becomes available.
System load is progressing towards normal levels and most users should find the email system operational. Until the system is fully restored, users who experience occasional timeouts and connection errors should refresh their connection.
Folder or Message Visibility Issue: Affected users have partial or no access to their email and/or folders are able to log in but do not see all expected folders or messages. Engineers are scanning system logs to detect these issues and compiling a list of affected users. We have prepared an automated fix to be applied following restoration of systems and a complete database cleanup.
Folder Deletion Issue: Affected users are unable to delete folders in Webmail, receiving the error Message: "The selected folder does not exist...." This issue is related to the Visibility Issue noted above and will be remediated in parallel.
"Lost" Folders Issue: Affected users note a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls". This issue will be addressed for many users during the weekend. If the issue persists following the weekend, please contact support.
POP/IMAP Setup & Webmail Last Page Issues: Affected users receive an error when trying to configure a mail client or navigate to the last page in Inbox in Webmail (error code undefined). These issues are related to the timeouts and connection errors noted above. Currently, most issues are resolved. Any remaining issues should be resolved once load returns to normal levels.
Message duplication issue: Affected users may experience a duplication of recently sent messages. This issue is a side effect of the system issue. Users can safely delete the duplicates. If manual deletion of a large number of duplicate messages is impractical, please contact support, as we can assist with deduplication once the system is fully restored and our maintenance is complete.
Unread Items Issue: For affected users, a number of messages or entire folder changed from "Read" to "Unread". Affected users may safely mark the messages as Read. We urge those users who find it impractical to mark a large number of messages as Read to contact support, as we can assist with this task once the system is fully restored and our maintenance is complete.
Engineers are staffed to work around the clock over the weekend to follow up on the issues from this week.
We apologize for these extended issues with the email platform and will post here as more information becomes available.
Impacted users have been identified and fixes will continue to be applied to resolve the following symptoms:
We apologize for these extended issues with the email platform and will post here as more information becomes available.
Progress is being made in making adjustments to provide relief to impacted users. While we make these changes, users may continue to experience the aforementioned symptoms.
These specific symptoms will resolve themselves once the issue has been resolved. We apologize for these extended issues with the email platform and will post here as more information becomes available.
We are continuing to monitor the environment and are making adjustments to provide relief to impacted users. While we make these changes, users may continue to experience the aforementioned symptoms.
These specific symptoms will resolve themselves once the issue has been resolved. We apologize for these extended issues with the email platform and will post here as more information becomes available.
We continue to monitor the environment as more users access the system throughout the day and are doing everything we can to drive system load down and improve responsiveness. Users may experience:
In addition, there are persistent issues with a small portion of mailboxes due to an emergency maintenance performed on 12 July 2017 at 8:00 pm CDT. Symptoms include, but are not limited to, the following:
These specific symptoms will resolve themselves once the issue has been resolved. We apologize for these extended issues with the email platform and will post here as more information becomes available.
We are continuing to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:
There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017.
These specific symptoms will resolve themselves once the issue has been resolved. We apologize for these extended issues with the email platform and will post here as more information becomes available.
At this time, message delivery times and webmail responsiveness have remained stable but we will continue to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:
There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 8:00 PM CDT 7/12/2017. While some of these symptoms may resolve themselves over time, if you are experiencing the below symptoms over a prolonged period of time, please contact Support (877.447.8936 or support@firespringcom) for further assistance. Symptoms include but are not limited to:
We are continuing to work to resolve the rest of the issues that some users may be experiencing. We will post more updates as they become available.
The email system and message delivery times have returned to normal levels. We will continue to monitor the environment as system load increases throughout the day.
As system load increases throughout the day users may experience:
We apologize for these extended issues with the email platform and will post here as more information becomes available.
Message delivery times have been restored to normal levels.
We continue to monitor system load and actively improve system performance across our Hosted Email environment.
During this increased system load, users may continue to see the following:
We apologize for these extended issues with the email platform and will post here as more information becomes available.
We continue to monitor system load and are actively improving system performance across our email system.
Until the issue is fully resolved, you may continue to see the following:
We apologize for these extended issues with the email platform and will post here as more information becomes available.
We continue to monitor system load and are actively improving performance across our email platform.
Until the issue is fully resolved, you may continue to see the following:
We apologize for these extended issues with the email platform and will post here as more information becomes available.
Our teams continue to actively work on reducing load and returning the system to normal operation. In the meantime, users affected by this may notice the following:
These specific symptoms will resolve themselves once the issue has been resolved. We will continue to post updates here as they become available.
Our teams are actively working to reduce load and return to normal operation. In the meantime, users affected by this may also experience the following:
These specific symptoms will resolve themselves once the issue has been resolved. We will post more information here as we know it.
Our Engineers are investigating some issues affecting the performance of the email system:
Some of the symptoms that are being reported would include:
We are working to resolve this issue as quickly as possible and will post more updates here as soon as we know them.
As part of our efforts to continually improve our product offerings, we will be performing a maintenance on our email platform. The maintenance window is scheduled to occur Wednesday, July 12th between 9:00 PM and 10:00 PM EDT.
While all measures are being taken to prevent any impact, you may experience a brief disruption of service during the maintenance window.