Config change silenced database health checks for 36 minutes
All GitHub.com services were fully inaccessible for all users worldwide for 36 minutes on the evening of August 14, 2024.
Database configuration changes need staged rollout and automated fast-rollback — a single bad config push that breaks health check responses can cause the routing layer to simultaneously drop all database hosts, producing a total outage with no code change involved.
An erroneous configuration change pushed to GitHub.com databases at 22:59 UTC caused the databases to stop responding to health check pings from the routing service.
The bad config caused database hosts to fail health checks → the routing service automatically marked them unhealthy → the production read-only database endpoint was removed from the pool → the application lost access to critical data for read operations → all GitHub.com services went down at 23:02 UTC; recovery required manually reverting the config change.
How would you design a database configuration change pipeline so that a single erroneous push cannot simultaneously make all hosts appear unhealthy to the routing layer?
