Past Incidents

The primary focus early this week was to establish stable mail delivery and login access. While we have made significant progress, some work remains. Our current focus is on individual mailbox repairs due to issues indirectly caused when the system was under high load. Now that the system is stabilized, these mailbox repairs are processing more quickly. If you believe that your mailbox is still having issues, please contact support so we can confirm that your issues are indeed related to the server issues from last week.

We continue to perform mailbox level cleanups to address the following issues, which we have identified proactively from system logs:

• mailboxes with login/access issues: this issue may cause intermittent errors for users that connect via POP or IMAP.
• extraneous folders prefixed with "lost-"
• missing messages or folders: the information is present, but not visible to email programs, including Webmail.

We recommend that those users who experience intermittent errors connecting via POP or IMAP utilize Webmail instead. While we are working to resolve all symptoms, our highest priority is ensuring all affected users have access to their mailbox.

Our initial pass of fixes for mailboxes that we detected as having login/access issues was successful. We will run a pass for newly discovered mailboxes that our logs have indicated are having issues. Once complete, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:

• missing messages or folders; the information is present but not visible to email programs including Webmail
• extraneous folders starting with "lost-"

As we continue to progress through our mailbox cleanup efforts, some users may see slowness within Webmail. Some mailboxes may require additional attention to resolve some of the issues experienced, and our team is actively working through those.

We are continuing to focus on maintaining access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. We have begun running a fix to address the issues for users that have not yet been able to successfully access their mailbox. During this time, some users may see some slowness within Webmail. Once completed, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:

• missing messages or folders; the information is present but not visible to email programs including Webmail
• extraneous folders starting with “lost-“

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

Our efforts this weekend have been focused on system wide cleanup to address the consequences of high system load after expanding capacity. These efforts are nearing completion. We will soon begin a more detailed mailbox-level cleanup effort to address some issues that we have seen among some users. While those updates happen there may be very brief periods where accessing email is slow or gives an error; trying again a few minutes later should be successful.

The mailbox level cleanup will address, in an automated fashion, the following issues for users we have been able to identify proactively:

• some users have not yet been able to successfully access their mailbox
• missing messages or folders; the information is present but not visible to email programs including Webmail
• extraneous folders starting with “lost-“

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those. We apologize for these extended issues with the email platform and will post here as more information becomes available.

We have completed another phase of repairs and have initiated the next phase. During this time, some users may experience sporadic delays along with Webmail, POP & IMAP connectivity issues.

Impacted users have been identified and fixes will continue to be applied to resolve the following symptoms:

• Unread Items Issue: A large number (or entire folder) of messages changed from "Read" to "Unread"
• Missing Subfolders Issue: Missing Subfolders removed without intervention. (These folders are hidden and not actually missing)
• "Lost" Folders Issue: a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls"
• POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to the last page in Inbox in Webmail (error code undefined).
• Folder Deletion Issue: Users unable to delete folders in Webmail. Error Message: "The selected folder does not exist...."
• Webmail Access Issue: Users unable to access their email through Webmail or Outlook, could receive error message web hard hat or "an error occurred ...", depending on the operation.
• Message duplication issue: Users may experience a duplication of recently sent messages.

We apologize for these extended issues with the email platform and will post here as more information becomes available.