Cloudflare/2022-06-21/~1.5h
A backbone change crushed 19 data centers
Jan 2022Dec 2022
19 of Cloudflare's busiest data centers dropped offline at once, hitting a large share of global traffic.
Lesson
Network and routing changes deserve the same staged rollout and review as application code. The "resilience project" is itself a risk window.
Trigger
A routing-policy change made during a long-running project to increase backbone resilience.
Mechanism
An incorrect change to how traffic was routed withdrew routes and concentrated load, taking the busiest sites down together — ironically, during resilience work.
Interview lens
Change management for network config; why infra changes need canaries too.
Recurring patterns
