Dual AWS region outage takes down notification dispatch
Notification dispatch was completely down for 18 minutes after two of PagerDuty's three datacenters failed simultaneously; the events-ingestion API stayed up throughout.
Redundancy plans must verify physical and network independence between supposedly separate failure domains, not just assume it from provider region boundaries.
A peering point shared by two AWS regions PagerDuty treated as independent datacenters degraded, taking both regions offline at once.
PagerDuty had designed for a single datacenter failure across three sites, but two of the three secretly shared a network peering point; when it degraded, both went down together, the dispatch cluster couldn't reach quorum under the resulting latency, and a coordinator misconfiguration slowed recovery further.
When asked to design a highly-available multi-region system, discuss how to verify true failure-domain independence between regions rather than trusting provider abstractions.
