43 seconds that cost 24 hours

Jan 2018Dec 2018

A day of stale, read-only metadata; webhooks and Pages builds frozen; manual write reconciliation afterward.

Lesson

Automated failover that reacts faster than your network heals can manufacture split-brain. Tune failover to application topology, not just node liveness.

Trigger

A 43-second network partition between the US-East hub and the primary data center during optical-equipment maintenance.

Mechanism

Orchestrator (Raft-based MySQL failover) promoted West-Coast primaries. When the link healed, East and West had each accepted writes the other lacked — divergence. Cross-country latency then made the app unusable.

Interview lens

The canonical CAP-in-practice story: consistency vs availability under partition, plus async-replication data-loss risk.

Recurring patterns