Failure class
CPU, memory, connections, file descriptors or disk run dry.
12 incidents on record
During a worldwide disruption of GitHub's collaboration, automation, and authentication services, roughly one in five web or API requests failed at peak, along with about half of archive and raw-file downloads.
Autoscaling and capacity monitoring must cover service-mesh sidecars and other infrastructure limits, while every retry path needs a bounded budget and backoff so recovery traffic cannot become a second outage.
For nearly seven hours in US-EAST-1, an internal Kinesis Data Streams cell degraded, raising latencies and error rates across CloudWatch Logs, S3 event delivery, Data Firehose, ECS, Lambda, Redshift, and Glue, with log backlogs taking until the next day to clear.
An automated placement or failover system validated against typical workloads can still misbehave on a skewed profile — test cluster-management automation against shard-count extremes, not just throughput extremes, and cap its reaction rate so a false health signal cannot trigger a redistribution storm.
A latent bug in Lambda's Frontend scaling logic degraded function invocations in one cell of US-EAST-1 for nearly four hours, causing elevated errors and latency across STS, Management Console, EKS cluster provisioning, Connect, and EventBridge.
Autoscaling systems can hide defects that only trigger past capacity levels never previously exercised in production, so scaling logic must be tested well beyond historical peak, not just up to it.
incident.io's API experienced intermittent 20-second timeouts over two weeks in early 2023, causing customers to see request failures whenever connection pool slots were fully occupied.
Instrument your database connection pool with per-operation duration metrics before an incident forces you to — connection pool exhaustion from many unnecessary short transactions is invisible without that data, and the root cause will look like a general slowdown with no obvious culprit.
incident.io experienced intermittent Postgres and Memcache connection timeouts across production nodes for approximately one week following their migration to GKE, affecting API response reliability for all customers.
After migrating to a new cloud platform, probe for hidden per-node networking behaviors (like Dataplane V2 CPU amplification on connection churn) before blaming application-layer connection pools — node-level packet drops can mimic dozens of unrelated app failures at once.
A near three-day full platform outage over Halloween weekend.
When one coordination layer underpins everything, its failure is total. Know the failure modes of your service-discovery/consensus substrate before you scale onto it.
GitHub.com experienced degraded service for a combined 8 hours 14 minutes across four events in late February 2020, with stalled writes on the mysql1 cluster affecting all authentication and core services that depended on it.
Your process manager may silently cap OS resource limits like LimitNOFILE to a value orders of magnitude below what you configured — always check the actual ulimits live in production processes, not just the config files.
Global 502s across ~10% of the web — Discord, Shopify, and Cloudflare's own dashboard went dark.
Config changes need the same staged rollout as code. "It's just a rule" is how a global outage starts.
Basecamp 3 was fully read-only for almost five hours, blocking all writes across the product — no new messages, todos, files, or edits — while reads continued to work.
When a hot, shared table nears a known integer key ceiling, treat the migration as urgent and monitor actual headroom, not just awareness that a fix is 'in progress.'
Linux builds were backed up or completely halted for roughly 17 hours across two days, affecting nearly all CircleCI customers' CI/CD pipelines during peak usage.
Investing in architecture to reduce a known failure mode isn't enough — also build and rehearse the operational tooling needed to intervene quickly when that failure mode recurs at larger scale.
Build throughput collapsed to roughly one transaction per minute; the backlog couldn't drain.
Recovery is a traffic event. The herd that hits you when an upstream comes back can be worse than its going down.
Basecamp and its sibling services were fully unreachable for 45 minutes and intermittently degraded for nearly two hours, affecting all customers on a Monday morning.
Your status page must be load-tested for outage-level traffic spikes, and incident communication must begin within five minutes of detection — not after internal triage is complete.