> For the complete documentation index, see [llms.txt](https://docs.tuannvm.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tuannvm.com/blog/reading/2025-10-23-summary-of-the-amazon-dynamodb-service-disruption-in-northern-virginia-us-east-1-region.md).

# 2025-10-23 Summary of the Amazon DynamoDB Service Disruption in Northern Virginia (US-EAST-1) Region

<https://aws.amazon.com/message/101925/>

Amazon Web Services experienced a **major service disruption** in the Northern Virginia (us-east-1) Region from **October 19, 2025, 11:48 PM PDT to October 20, 2025, 2:20 PM PDT**. The outage impacted DynamoDB, EC2, Network Load Balancers (NLB), and multiple dependent AWS services.

#### Key Points

1. **Primary Cause**
   * A **latent race condition** in DynamoDB’s **DNS management system** caused an **empty DNS record** for the regional endpoint, preventing new connections.
   * Manual operator intervention was required because the system entered an **inconsistent state** that DNS automation could not resolve.
2. **Service Impact Timeline**
   * **DynamoDB (11:48 PM – 2:40 AM)**
     * DNS resolution failures caused API errors and prevented new connections.
     * Global table replication continued but with delays to/from us-east-1.
     * Full recovery at 2:40 AM.
   * **EC2 (11:48 PM – 1:50 PM)**
     * Instance launches failed due to **DWFM (DropletWorkflow Manager) lease expirations** caused by failed DynamoDB-dependent state checks.
     * Network configuration propagation was delayed, leading to connectivity issues. Full recovery at 1:50 PM.
   * **Network Load Balancer (5:30 AM – 2:09 PM)**
     * **Health check failures** occurred because new EC2 instances lacked network configuration.
     * Triggered failovers, DNS removals, and connection errors. Stabilized after disabling auto failover.
   * **Other Services**
     * **Lambda, ECS/EKS, Fargate, STS, Redshift, and Amazon Connect** experienced errors due to dependencies on DynamoDB, EC2 launches, and NLB health checks.
     * Most services recovered by early afternoon, with Redshift cluster availability fully restored by **October 21, 4:05 AM PDT**.
3. **Remediation and Future Improvements**
   * DynamoDB DNS automation globally disabled until the race condition is fixed and safeguards are added.
   * EC2 will implement **better throttling and DWFM recovery tests**.
   * NLB will add **velocity controls** for AZ failover events.
   * AWS is enhancing **scale testing** and **cross-service dependency resilience** to shorten recovery times.

***

#### Mermaid Sequence Diagram of the Incident

{% @mermaid/diagram content="sequenceDiagram

participant DDB as DynamoDB

participant EC2 as EC2

participant NLB as NLB

participant SVC as Services

DDB->>DDB: Empty DNS record

SVC->>DDB: API fails

EC2->>DDB: State check fails

EC2-->>EC2: New instance fail

Note over DDB,EC2: DNS fixed

EC2->>EC2: Restores leases

NLB->>EC2: Health checks fail

NLB-->>SVC: Errors impact services

Note over NLB,SVC: NLB errors persist

SVC->>SVC: Throttling

Note over DDB,EC2: Full recovery

Note over NLB,SVC: Full recovery" %}

This incident demonstrates the **cascading impact of interdependent cloud services** when a single DNS automation failure affects core APIs, compute instances, and network load balancers across multiple services.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.tuannvm.com/blog/reading/2025-10-23-summary-of-the-amazon-dynamodb-service-disruption-in-northern-virginia-us-east-1-region.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.