Amazon Web Services (AWS) has explained the hours-long service disruption that caused many websites and Internet-connected services to go offline earlier this week.

The Amazon Simple Storage Service (S3) team was debugging a problem in the S3 billing system on Tuesday morning when one team member "executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process," Amazon wrote in a post-mortem describing the incident. That's when things went wrong. "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems."

An index subsystem that "manages the metadata and location information of all S3 objects in the [Virginia data center] region" was one of the two affected, Amazon wrote. "This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects."

Amazon had to do a full restart of the affected systems after the unexpected removal of capacity. "While these subsystems were being restarted, S3 was unable to service requests," Amazon wrote. "Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable."

Preventing a recurrence

Although S3 subsystems are designed to keep working "with little or no customer impact" even when significant capacity fails or is removed, Amazon said it had not "completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected."

Amazon says it has a way to prevent this problem from recurring. Amazon has to be able to remove capacity when necessary, but "the tool used allowed too much capacity to be removed too quickly," the company said. "We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level." Amazon is also auditing other operational tools to ensure that they have similar safety checks and will "make changes to improve the recovery time of key S3 subsystems."

Amazon service update system went down, too

Amazon itself relies on S3 and was thus temporarily unable to provide public service updates on the AWS Service Health Dashboard. Going forward, the dashboard administration console will utilize multiple data center regions in order to stay online when one region is disrupted. That means if a data center in Virginia fails, the system could still operate from data centers elsewhere in the US or world.

Apparent victims of the outage included The AV Club, Trello, Quora, IFTTT, Open Whisper Systems, and websites created with Wix. Some people also reported problems with Internet-connected devices such as an oven, remote light controllers (including one powered by IFTTT), and a front gate.

"[W]e want to apologize for the impact this event caused for our customers," Amazon concluded. "While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses. We will do everything we can to learn from this event and use it to improve our availability even further."