Database contention triggers day-long build queue collapse
Linux builds were backed up or completely halted for roughly 17 hours across two days, affecting nearly all CircleCI customers' CI/CD pipelines during peak usage.
Investing in architecture to reduce a known failure mode isn't enough — also build and rehearse the operational tooling needed to intervene quickly when that failure mode recurs at larger scale.
Rising build demand at peak traffic exposed database contention that prevented builds from being dequeued, and the standard scaling tools had no effect on the growing backlog.
Queued database operations backed up faster than mitigation attempts could isolate the cause, forcing a failover to a new primary to kill stuck operations; once builds were flowing again, a separate bug blocked builds from ever reaching the runnable state, and the built-in failure-backoff throttles misfired and slowed recovery further, requiring engineers to build ad hoc tooling live to manually push builds through in batches.
When asked how to prevent a shared database from becoming a single point of contention under load, discuss splitting hot datasets into isolated stores and building dedicated incident-response tooling ahead of time, not during the outage.
