AWS/2021-12-07/~5h
The autoscaler that fought the network
Jan 2021Dec 2021
A major us-east-1 disruption affecting a long list of AWS services and the customers built on them.
Lesson
Feedback loops between automated clients and the network they depend on cause congestion collapse. Monitoring must not share fate with the thing it monitors.
Trigger
Automated scaling activity on the main AWS network provoked unexpected behavior from a large number of internal clients.
Mechanism
A surge of connection activity overwhelmed networking devices between the internal and main networks. Retries plus the impaired network created a congestion-collapse feedback loop — and monitoring rode the same impaired path, blinding responders.
Interview lens
Congestion collapse, backoff/jitter on retries, and observability that survives the outage it's reporting on.
Recurring patterns
