One customer's config, the whole CDN down

Jan 2021Dec 2021

85% of Fastly's network returned errors within a minute — gov.uk, Reddit, Amazon and major news sites went down together.

Lesson

A latent bug can sit quiet until ordinary input you don't control wakes it. That input is part of your test matrix.

Trigger

A valid customer configuration change tripped a latent bug that had been introduced in an earlier software deployment.

Mechanism

The dormant bug only woke when a particular, entirely legitimate customer setting hit it — at which point most of the global network began failing simultaneously.

Interview lens

Talk about latent bugs surfaced by input, and why you fuzz/replay real customer configs in staging.

Recurring patterns