CircleCI/2015-01-01/hours
When GitHub came back, the queue fell over
Jan 2015Dec 2015
Build throughput collapsed to roughly one transaction per minute; the backlog couldn't drain.
Lesson
Recovery is a traffic event. The herd that hits you when an upstream comes back can be worse than its going down.
Trigger
GitHub recovered from its own outage, and all the backed-up webhooks and builds hit CircleCI at once.
Mechanism
The recovery surge pushed CircleCI's queue subsystem into a pathological slow state. With throughput near zero, the flood of queued work had nowhere to go.
Interview lens
Designing for upstream-recovery surges: backpressure, rate limits, and draining strategies.
Recurring patterns
