Summary of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region

On Monday, Oct 20, AWS had a massive failure that took many services across the Internet offline. The core issue was a failure in DNS automation.

Many of the largest AWS services rely extensively on DNS to provide seamless scale, fault isolation and recovery, low latency, and locality. Services like DynamoDB maintain hundreds of thousands of DNS records to operate a very large heterogeneous fleet of load balancers in each Region. Automation is crucial to ensuring that these DNS records are updated frequently to add additional capacity as it becomes available, to correctly handle hardware failures, and to efficiently distribute traffic to optimize customers' experience. […] The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service's regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair.

Without DNS no system knows how to find any other systems. Additionally AWS automation for many other services relies on DynamoDB. This was a particularly bad failure state.