Issue Summary:



On June 28, 2019 at 4:30 a.m. PDT some of our servers became unavailable, causing degraded performance in our job processing system. This resulted in delays or errors with features such notifications, unfurls, and message posting.



During this time, approximately 10-25% of jobs resulted in errors or failure. By 10:00 a.m. PDT, we fully restored message delivery and reduced the error rates to less than 5% as the team continued to work on a full recovery.



At 1:05 p.m. PDT, a separate issue increased server load and dropped a large number of user connections. Reconnection attempts further increased the server load, slowing down customer reconnection. Server capacity was freed up eventually, enabling all customers to reconnect by 1:36 p.m. PDT.



Full service restoration was completed by 7:20 p.m. PDT. During this period, customers faced delays or failure with a number of features including file uploads, notifications, search indexing, link unfurls, and reminders.



Now that service has been restored, the response team is continuing their investigation and working to calculate service interruption time as soon as possible. We’re also working on preventive measures to ensure that this doesn’t happen again in the future. If you’re still running into any issues, please reach out to us at feedback@slack.com.

A fix has been rolled out and all features should be working normally. We are immensely sorry for the disruption to your work day and are continuing the work to make sure this doesn't happen again.



If you are still seeing any issues, please don't hesitate to drop us a line via our help page or feedback@slack.com — we're here to help.

We're seeing positive signs with some changes being rolled out, and are continuing to work to get things back to 100%. We really appreciate your patience.

Customers may see degraded functionality for several other features, including notifications, reminders and workspace member lists — we're working to restore that. File uploads should now work after a refresh (Cmd/Ctrl + R).

We’re still working hard to restore file upload functionality. Thank you for your continued patience.

File uploads are still not working for some users but refreshing Slack (Cmd/Ctrl + R) may restore functionality.

We don't have any updates right now. Refreshing Slack (Cmd/Ctrl + R) should restore connectivity but we’re still working on file uploads.

We expect users to be able to connect to Slack after a refresh (Cmd/Ctrl + R) but we are still experiencing some issues with file uploads.

We are still seeing reports from many users unable to connect to Slack. We truly appreciate your patience while we monitor the situation.

Some users have reported problems loading Slack. We'll continue to update you on our progress.

Some users have reported intermittent connectivity issues. We remain hard at work to come to a full resolution.

Still no news to share on a resolution, but please continue to let us know if you're running into any trouble after reloading. We'll return with more updates soon.

We're still working on getting things completely fixed up, but for now reloading should help. If this doesn't lead to improvements, though, drop us a line at feedback@slack.com.

We’re seeing that functionality may be restored after reloading (Cmd/Ctrl + R). If you're still running into trouble after giving that a try, please let us know at feedback@slack.com.

No news to share just yet, but we remain hard at work to get everything running smoothly again. We appreciate your patience in the meantime.

We're seeing some signs of recovery, though we aren't in the clear yet. We'll continue to keep you posted on our progress.

We're still investigating some potential fixes, but don't have any news to share just yet. Thank you for your continued patience.

We're still on the case, and are doing everything we can to restore all functionality in Slack. We'll be back with more updates soon.

We're continuing to work on getting everything up and running again. Though we saw signs of improvement earlier, unfortunately the fix we had in mind didn't do the trick. We're now looking into another possible fix and will share updates as they become available.

We're still looking into the issues with various Slack services, and will continue to keep you posted on our progress. Apologies again for the disruption to your day.

We have no more news to share just yet, just that we've got all eyes on this. We'll keep you posted with updates as we have them.

Work is still ongoing to get functionality back to normal. We appreciate you sticking with us.

Functionality is almost back to normal, but some users may see some sporadic problems while we work on remediation.

Many of our users may still be having problems with several Slack services. We have all hands on-deck working on a solution. We’ll share more news with you all shortly.

We’re still figuring out the problem some users are having with multiple Slack services. We’re so sorry for the interruption. When we find out more, we’ll let you know.

We’re still investigating the issues with several Slack services, but we don’t have additional information just yet. We’ll be back soon with another update. Sorry for the interruption.

Some customers are still reporting issues with multiple Slack services. We're currently investigating and will keep you posted as soon as we have more information. We're sorry for the disruption.

We received reports of multiple issues regarding Slack's degraded performance, which we are currently investigating.