Database Migration Cascades Into Site-Wide DNS Outage
A single-site DNS outage degraded IDE code completions for 4% of Copilot users, delayed 25% of Actions workflows by over 5 minutes, and caused a complete code search outage for roughly 4 hours.
A quick mitigation for a localized infra failure can silently break cross-site connectivity elsewhere, so any DNS or routing fix should be validated for its effect on the broader topology before being treated as the resolution.
A database migration at one of GitHub's sites caused the local DNS infrastructure to stop resolving lookups.
Engineers' attempts to recover the affected database triggered cascading failures in the site's DNS systems; a first mitigation that repointed the degraded site's DNS to a healthy site restored local resolution but broke return connectivity from healthy sites back to the degraded one, forcing the team to devise and deploy a second, temporary DNS resolution mechanism before full recovery.
When asked to design multi-site DNS infrastructure, discuss how to isolate a single site's DNS failure from database maintenance and how to validate that failover mitigations don't break connectivity between sites.
