AWS/2021-12-07/~5h
The autoscaler that fought the network
Jan 2021Dec 2021
A major us-east-1 disruption affecting a long list of AWS services and the customers built on them.
Lesson
Feedback loops between automated clients and the network they depend on cause congestion collapse. Monitoring must not share fate with the thing it monitors.
Trigger
Automated scaling activity on the main AWS network provoked unexpected behavior from a large number of internal clients.
Mechanism
A surge of connection activity overwhelmed networking devices between the internal and main networks. Retries plus the impaired network created a congestion-collapse feedback loop — and monitoring rode the same impaired path, blinding responders.
Interview lens
Retries can deepen an overload. Wait longer between attempts, spread them out, and keep monitoring available during the outage.
Recurring patterns
