Service discovery collapse took down all systems for 10 hours
The US region of Datadog was degraded for roughly 10 hours on September 24–25, 2020, with the web tier hitting 60–90% error rates, leaving customers unable to reliably access dashboards, alerts, APM, or infrastructure monitoring.
Service discovery and configuration systems must serve stale data gracefully when overloaded rather than failing hard — any system that grapples for a central dependency at startup turns a localized failure into a platform-wide thundering herd.
A routine recycling of a small latency-measuring cluster caused its service discovery entries to disappear, prompting the intake cluster — which had been silently misconfigured a month earlier to use the local DNS resolver instead of a static file — to flood service discovery with non-cached NXDOMAIN queries at 10× normal volume within seconds.
The NXDOMAIN flood overwhelmed the service discovery cluster, which lost quorum → once node-level DNS caches expired, every service across the platform joined a thundering herd hammering the same broken cluster → service discovery could no longer register services or answer queries → services failed to locate dependencies or load runtime config at startup → the web tier sustained 60–90% error rates; recovery required manually isolating the cluster, controlling re-admission, and iteratively removing hard dependencies service by service.
How would you design a service discovery system so that a thundering herd of reconnecting clients after an outage cannot cause the discovery cluster itself to lose quorum and extend the outage?
