Puppet bug corrupted DNS and cascaded into fileserver exhaustion
GitHub.com was fully unavailable for 42 minutes on January 8, 2014, with an additional 95 minutes of degraded access for a subset of repositories while fileservers were triaged and restored.
Deployment systems that regenerate critical infrastructure config (like DNS zone files) must validate API responses for completeness before applying them — and must never depend on the very service they are rebuilding.
A Puppet manifest bug caused only the authoritative name server to restart after a firewall and DNS config change, leaving the caching name server querying an IP that was no longer serving DNS.
Caching name server timed out on DNS queries → a triggered deployment attempted to rebuild the zone file via an API call that itself required working DNS → the broken DNS caused the API to return partial data → the zone file was generated with records stripped → mass NXDOMAIN responses; simultaneously the DNS outage caused a process storm on fileservers → memory exhaustion → routing layer back-pressure blocked access to healthy fileservers too.
How would you design a DNS zone file generation pipeline to be safe when the DNS infrastructure it depends on is partially degraded?
