73 hours down via one coordination layer

Jan 2021Dec 2021

A near three-day full platform outage over Halloween weekend.

Lesson

When one coordination layer underpins everything, its failure is total. Know the failure modes of your service-discovery/consensus substrate before you scale onto it.

Trigger

A new streaming feature on the HashiCorp Consul cluster, combined with an unusual load pattern, triggered heavy contention in Consul's write path.

Mechanism

Consul underpinned service discovery and health for everything. Under contention its performance collapsed; a separate Consul bug then blocked recovery. Because so many services depended on it, the whole platform stayed down for days.

Interview lens

Service discovery can become a single point of failure. Isolate it and test what happens when it goes down.

Recurring patterns