# Summary of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region

<https://aws.amazon.com/message/101925/>

- Host: [aws.amazon.com](https://links.thingelstad.xyz/hosts/aws.amazon.com/)
- Saved: October 23, 2025 (2025-10-24T03:21:16Z)
- Record: https://links.thingelstad.xyz/links/f8bf443a8165/
- JSON: https://links.thingelstad.xyz/links/f8bf443a8165.json

## What Jamie wrote

On Monday, Oct 20, AWS had a massive failure that took many services across the Internet offline. The core issue was a failure in DNS automation.

> Many of the largest AWS services rely extensively on DNS to provide seamless scale, fault isolation and recovery, low latency, and locality. Services like DynamoDB maintain hundreds of thousands of DNS records to operate a very large heterogeneous fleet of load balancers in each Region. Automation is crucial to ensuring that these DNS records are updated frequently to add additional capacity as it becomes available, to correctly handle hardware failures, and to efficiently distribute traffic to optimize customers' experience. […] **The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service's regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair.**

Without DNS no system knows how to find any other systems. Additionally AWS automation for many other services relies on DynamoDB. This was a particularly bad failure state.

## Facts

- Kind: [Page](https://links.thingelstad.xyz/kinds/page/)

## Appears in

- [WT331: Weekly Thing 331 / RFC, Security, Tokens](https://weekly.thingelstad.com/archive/331/), Notable

## Same host

- [Introducing Amazon Bedrock AgentCore: Securely deploy and operate AI agents at any scale (preview) | AWS News Blog](https://links.thingelstad.xyz/links/92c00f498539/) (aws.amazon.com, July 17, 2025)
- [Introducing AWS Lambda Managed Instances: Serverless simplicity with EC2 flexibility | AWS News Blog](https://links.thingelstad.xyz/links/74d34753a948/) (aws.amazon.com, December 4, 2025)
- [Amazon S3 Vectors now generally available with increased scale and performance | AWS News Blog](https://links.thingelstad.xyz/links/10a2424966c4/) (aws.amazon.com, December 4, 2025)
- [Introducing OpenClaw on Amazon Lightsail to run your autonomous private AI agents | AWS News Blog](https://links.thingelstad.xyz/links/eb3bc6487a1f/) (aws.amazon.com, March 4, 2026)
- [Four security principles for agentic AI systems | AWS Security Blog](https://links.thingelstad.xyz/links/d79a0cb89f95/) (aws.amazon.com, April 9, 2026)

## Saved the same week

- [Scripts I wrote that I use all the time](https://links.thingelstad.xyz/links/db6aabd7a5b7/) (evanhahn.com, October 23, 2025)
- [Disco - Self-hosting that doesn't feel like self-hosting](https://links.thingelstad.xyz/links/4bb00881134e/) (disco.cloud, October 23, 2025)
- [Measuring Engineering Productivity - Off By One](https://links.thingelstad.xyz/links/9b2e1ea827a5/) (justoffbyone.com, October 23, 2025)
- [Space Exploration Logo Archive](https://links.thingelstad.xyz/links/0df3519b85ac/) (spaceexplorationlogoarchive.webflow.io, October 23, 2025)
- [Rivian’s Also e-bike is like nothing you’ve ever seen before | The Verge](https://links.thingelstad.xyz/links/3abc39b01fb7/) (theverge.com, October 23, 2025)

## Elsewhere

- Wayback Machine: https://web.archive.org/web/*/https://aws.amazon.com/message/101925/