Database Migration Cascades Into Site-Wide DNS Outage
A single-site DNS outage degraded IDE code completions for 4% of Copilot users, delayed 25% of Actions workflows by over 5 minutes, and caused a complete code search outage for roughly 4 hours.
A quick mitigation for a localized infra failure can silently break cross-site connectivity elsewhere, so any DNS or routing fix should be validated for its effect on the broader topology before being treated as the resolution.
A database migration at one of GitHub's sites caused the local DNS infrastructure to stop resolving lookups.
Engineers' attempts to recover the affected database triggered cascading failures in the site's DNS systems; a first mitigation that repointed the degraded site's DNS to a healthy site restored local resolution but broke return connectivity from healthy sites back to the degraded one, forcing the team to devise and deploy a second, temporary DNS resolution mechanism before full recovery.
Keep each site's DNS independent from database maintenance, and test that failover still preserves connectivity between sites.
