Failure class
A synchronized surge — retries, reconnects, cache stampede — buries a resource.
8 incidents on record
Signed-out users experienced 504 errors (peaking at 34% error rate) for pull requests, issues, and releases across GitHub.com for two hours; authenticated users were unaffected.
Isolate unauthenticated traffic to separate backend resources or rate-limit public endpoints independently to prevent malicious traffic surges from degrading service for authenticated users.
Self-hosted and larger GitHub Actions runners worldwide could not connect for nearly 5 hours, delaying or failing workflow jobs, while the resulting reconnection surge briefly degraded API Requests, Issues, and Pages with added latency and elevated error rates.
Certificate expiration is a fully predictable failure mode, so renewal must be automated with independent expiry monitoring, and reconnect paths need circuit breakers so a recovering fleet of clients doesn't stampede shared APIs on its way back up.
Cloudflare's dashboard was fully unavailable and its APIs experienced two separate periods of severe degradation for roughly 75 minutes, affecting all customers attempting to make configuration changes.
Services sitting in the critical authorization path for all API requests need staged rollouts with automatic rollback and substantial capacity headroom — a single client-side loop bug can generate enough traffic to collapse authentication for an entire platform.
Widespread connection failures on the first work-Monday of the year, exactly when everyone logged on at once.
Demand can outrun autoscaling. Pre-scale for known surges and make health checks resilient to transient loss so they don't amplify the failure.
The US region of Datadog was degraded for roughly 10 hours on September 24–25, 2020, with the web tier hitting 60–90% error rates, leaving customers unable to reliably access dashboards, alerts, APM, or infrastructure monitoring.
Service discovery and configuration systems must serve stale data gracefully when overloaded rather than failing hard — any system that grapples for a central dependency at startup turns a localized failure into a platform-wide thundering herd.
Poland's largest e-commerce site went down at the moment a campaign was driving peak traffic.
Capacity you can't actually schedule isn't capacity. Test that autoscaling can really claim idle resources under real load.
Build throughput collapsed to roughly one transaction per minute; the backlog couldn't drain.
Recovery is a traffic event. The herd that hits you when an upstream comes back can be worse than its going down.
The Supermarket community site crashed two hours after launch with intermittent unresponsiveness.
Aggressive health checks turn a latency blip into an outage. Set timeouts comfortably above your real tail latency.