Lambda Frontend Scaling Bug Breaks US-EAST-1
A latent bug in Lambda's Frontend scaling logic degraded function invocations in one cell of US-EAST-1 for nearly four hours, causing elevated errors and latency across STS, Management Console, EKS cluster provisioning, Connect, and EventBridge.
Autoscaling systems can hide defects that only trigger past capacity levels never previously exercised in production, so scaling logic must be tested well beyond historical peak, not just up to it.
The Lambda Frontend fleet auto-scaled past a capacity threshold in a single cell that had never been reached before, exposing a dormant software defect.
Crossing the new threshold caused execution environments to be allocated but never actually put to use by the Frontend, so no usable capacity existed to serve incoming invocations; synchronous calls errored out while asynchronous and streaming events piled up in a backlog, and dependent AWS services degraded in turn until engineers scaled the fleet back below the threshold.
When asked about safe autoscaling design, discuss why untested scale thresholds are as risky as untested code paths, and how cellular architectures contain blast radius when a cell fails.
