systemsfailed.dev https://www.systemsfailed.dev/ Real engineering postmortems, organised by how they failed. en Anthropic — Claude requests routed to the wrong context servers https://www.systemsfailed.dev/incident/anthropic-2025-context-window-routing https://www.systemsfailed.dev/incident/anthropic-2025-context-window-routing Sat, 08 Aug 2026 00:00:00 GMT Degraded Sonnet 4 responses initially affected 0.8% of requests, reached 16% in the worst hour, and touched about 30% of active Claude Code users at least once. Cruise — The collision logic that chose to keep moving https://www.systemsfailed.dev/incident/cruise-2023-post-collision-pullover https://www.systemsfailed.dev/incident/cruise-2023-post-collision-pullover Sat, 08 Aug 2026 00:00:00 GMT A driverless Cruise vehicle pulled a pedestrian forward after contact; Cruise paused its driverless fleet 24 days later and recalled the collision-detection software installed on 950 automated-driving systems. GitHub — A feature flag blocked every Copilot request https://www.systemsfailed.dev/incident/github-2025-copilot-rate-limiter https://www.systemsfailed.dev/incident/github-2025-copilot-rate-limiter Sat, 08 Aug 2026 00:00:00 GMT Most GitHub Copilot features were degraded for 25 minutes, from 17:55 to 18:20 UTC, as the global rate limiter returned HTTP 403 responses for every request affected by the invalid configuration. Google — AI Overviews grounded answers in satire and trolls https://www.systemsfailed.dev/incident/google-2024-ai-overviews-data-voids https://www.systemsfailed.dev/incident/google-2024-ai-overviews-data-voids Sat, 08 Aug 2026 00:00:00 GMT Some US Search users received odd, inaccurate, or unhelpful AI Overviews, including answers grounded in satire, sarcastic forum posts, and misleading user-generated advice, even though policy-violating responses were rare overall. OpenAI — The reward signal that made GPT-4o sycophantic https://www.systemsfailed.dev/incident/openai-2025-gpt4o-sycophancy https://www.systemsfailed.dev/incident/openai-2025-gpt4o-sycophancy Sat, 08 Aug 2026 00:00:00 GMT ChatGPT became markedly over-agreeable, sometimes validating doubts, fueling anger, reinforcing negative emotions, or encouraging impulsive actions until OpenAI restored the previous GPT-4o version. AWS — Lambda Frontend Scaling Bug Breaks US-EAST-1 https://www.systemsfailed.dev/incident/aws-2023-lambda-frontend-scaling https://www.systemsfailed.dev/incident/aws-2023-lambda-frontend-scaling Sat, 25 Jul 2026 00:00:00 GMT A latent bug in Lambda's Frontend scaling logic degraded function invocations in one cell of US-EAST-1 for nearly four hours, causing elevated errors and latency across STS, Management Console, EKS cluster provisioning, Connect, and EventBridge. GitHub — Database Migration Cascades Into Site-Wide DNS Outage https://www.systemsfailed.dev/incident/github-2024-dns-database-migration https://www.systemsfailed.dev/incident/github-2024-dns-database-migration Sat, 25 Jul 2026 00:00:00 GMT A single-site DNS outage degraded IDE code completions for 4% of Copilot users, delayed 25% of Actions workflows by over 5 minutes, and caused a complete code search outage for roughly 4 hours. GitHub — Expired internal SSL cert breaks Actions runner connectivity https://www.systemsfailed.dev/incident/github-2026-actions-runner-cert-expiry https://www.systemsfailed.dev/incident/github-2026-actions-runner-cert-expiry Sat, 25 Jul 2026 00:00:00 GMT Self-hosted and larger GitHub Actions runners worldwide could not connect for nearly 5 hours, delaying or failing workflow jobs, while the resulting reconnection surge briefly degraded API Requests, Issues, and Pages with added latency and elevated error rates. GitHub — Public endpoints overwhelmed by abusive traffic https://www.systemsfailed.dev/incident/github-2026-abusive-traffic-504 https://www.systemsfailed.dev/incident/github-2026-abusive-traffic-504 Tue, 14 Jul 2026 00:00:00 GMT Signed-out users experienced 504 errors (peaking at 34% error rate) for pull requests, issues, and releases across GitHub.com for two hours; authenticated users were unaffected. AWS — Kinesis cell manager mistakes healthy hosts for dead https://www.systemsfailed.dev/incident/aws-2024-kinesis-cell-management-shard-storm https://www.systemsfailed.dev/incident/aws-2024-kinesis-cell-management-shard-storm Fri, 10 Jul 2026 00:00:00 GMT For nearly seven hours in US-EAST-1, an internal Kinesis Data Streams cell degraded, raising latencies and error rates across CloudWatch Logs, S3 event delivery, Data Firehose, ECS, Lambda, Redshift, and Glue, with log backlogs taking until the next day to clear. Basecamp — Primary key overflow on events table forces read-only mode https://www.systemsfailed.dev/incident/basecamp-2018-events-table-integer-overflow https://www.systemsfailed.dev/incident/basecamp-2018-events-table-integer-overflow Sat, 04 Jul 2026 00:00:00 GMT Basecamp 3 was fully read-only for almost five hours, blocking all writes across the product — no new messages, todos, files, or edits — while reads continued to work. CircleCI — Database contention triggers day-long build queue collapse https://www.systemsfailed.dev/incident/circleci-2015-database-overload-build-queue-collapse https://www.systemsfailed.dev/incident/circleci-2015-database-overload-build-queue-collapse Sat, 04 Jul 2026 00:00:00 GMT Linux builds were backed up or completely halted for roughly 17 hours across two days, affecting nearly all CircleCI customers' CI/CD pipelines during peak usage. CircleCI — Backwards-incompatible schema change stalls all job execution https://www.systemsfailed.dev/incident/circleci-2021-backwards-incompatible-schema-change https://www.systemsfailed.dev/incident/circleci-2021-backwards-incompatible-schema-change Sat, 04 Jul 2026 00:00:00 GMT All job executors were blocked from running customer jobs for roughly 53 minutes, with elevated queue times continuing for machine executors for over an hour longer, affecting all CircleCI customers. PagerDuty — Dual AWS region outage takes down notification dispatch https://www.systemsfailed.dev/incident/pagerduty-2013-dual-region-peering-outage https://www.systemsfailed.dev/incident/pagerduty-2013-dual-region-peering-outage Sat, 04 Jul 2026 00:00:00 GMT Notification dispatch was completely down for 18 minutes after two of PagerDuty's three datacenters failed simultaneously; the events-ingestion API stayed up throughout. AWS — Config change killed Seoul region DNS for 84 minutes https://www.systemsfailed.dev/incident/aws-2018-seoul-dns-resolver https://www.systemsfailed.dev/incident/aws-2018-seoul-dns-resolver Sat, 27 Jun 2026 00:00:00 GMT EC2 instances in the AWS AP-NORTHEAST-2 (Seoul) region were unable to resolve DNS for 84 minutes, affecting all customers running workloads in that region. Cloudflare — Dashboard bug overwhelmed auth API for 75 minutes https://www.systemsfailed.dev/incident/cloudflare-2025-tenant-api-thundering-herd https://www.systemsfailed.dev/incident/cloudflare-2025-tenant-api-thundering-herd Sat, 27 Jun 2026 00:00:00 GMT Cloudflare's dashboard was fully unavailable and its APIs experienced two separate periods of severe degradation for roughly 75 minutes, affecting all customers attempting to make configuration changes. Datadog — Service discovery collapse took down all systems for 10 hours https://www.systemsfailed.dev/incident/datadog-2020-service-discovery-thundering-herd https://www.systemsfailed.dev/incident/datadog-2020-service-discovery-thundering-herd Sat, 27 Jun 2026 00:00:00 GMT The US region of Datadog was degraded for roughly 10 hours on September 24–25, 2020, with the web tier hitting 60–90% error rates, leaving customers unable to reliably access dashboards, alerts, APM, or infrastructure monitoring. GitHub — Puppet bug corrupted DNS and cascaded into fileserver exhaustion https://www.systemsfailed.dev/incident/github-2014-dns-puppet-cascade https://www.systemsfailed.dev/incident/github-2014-dns-puppet-cascade Sat, 27 Jun 2026 00:00:00 GMT GitHub.com was fully unavailable for 42 minutes on January 8, 2014, with an additional 95 minutes of degraded access for a subset of repositories while fileservers were triaged and restored. GitHub — ProxySQL file descriptor limit silently capped by system process manager https://www.systemsfailed.dev/incident/github-2020-proxysql-file-descriptor-exhaustion https://www.systemsfailed.dev/incident/github-2020-proxysql-file-descriptor-exhaustion Sat, 27 Jun 2026 00:00:00 GMT GitHub.com experienced degraded service for a combined 8 hours 14 minutes across four events in late February 2020, with stalled writes on the mysql1 cluster affecting all authentication and core services that depended on it. incident.io — Database upgrade made incident IDs skip 32 numbers https://www.systemsfailed.dev/incident/incident-io-2021-postgres-sequence-skip https://www.systemsfailed.dev/incident/incident-io-2021-postgres-sequence-skip Sat, 27 Jun 2026 00:00:00 GMT Multiple customer organizations saw their incident identifiers jump by up to 32 (e.g. from #INC-20 to #INC-52), eroding trust in the numbering system, though no data was lost or misattributed. incident.io — Unnecessary Slack transactions starved database connection pool https://www.systemsfailed.dev/incident/incident-io-2023-connection-pool-exhaustion https://www.systemsfailed.dev/incident/incident-io-2023-connection-pool-exhaustion Sat, 27 Jun 2026 00:00:00 GMT incident.io's API experienced intermittent 20-second timeouts over two weeks in early 2023, causing customers to see request failures whenever connection pool slots were fully occupied. incident.io — GKE Dataplane V2 CPU starvation from concurrent TCP storms https://www.systemsfailed.dev/incident/incident-io-2023-gke-dataplane-tcp-storms https://www.systemsfailed.dev/incident/incident-io-2023-gke-dataplane-tcp-storms Sat, 27 Jun 2026 00:00:00 GMT incident.io experienced intermittent Postgres and Memcache connection timeouts across production nodes for approximately one week following their migration to GKE, affecting API response reliability for all customers. incident.io — AWS us-east-1 outage cascaded through four hidden third-party dependencies https://www.systemsfailed.dev/incident/incident-io-2025-aws-outage-cascade https://www.systemsfailed.dev/incident/incident-io-2025-aws-outage-cascade Sat, 27 Jun 2026 00:00:00 GMT incident.io's on-call notifications were delayed for over two hours, SAML authentication was intermittently unavailable, Scribe's AI note-taker was offline for ~5 hours, and the deployment pipeline was blocked — all driven by AWS us-east-1 dependency propagation through their telecom provider, auth provider, transcription provider, and Docker Hub. GitHub — Config change silenced database health checks for 36 minutes https://www.systemsfailed.dev/incident/github-2024-database-config-outage https://www.systemsfailed.dev/incident/github-2024-database-config-outage Fri, 26 Jun 2026 00:00:00 GMT All GitHub.com services were fully inaccessible for all users worldwide for 36 minutes on the evening of August 14, 2024. Basecamp — Extortion DDoS took Basecamp down for 45 minutes https://www.systemsfailed.dev/incident/basecamp-2014-ddos-extortion https://www.systemsfailed.dev/incident/basecamp-2014-ddos-extortion Thu, 25 Jun 2026 00:00:00 GMT Basecamp and its sibling services were fully unreachable for 45 minutes and intermittently degraded for nearly two hours, affecting all customers on a Monday morning. Allegro — A marketing campaign DDoSed itself https://www.systemsfailed.dev/incident/allegro-2018-scaling https://www.systemsfailed.dev/incident/allegro-2018-scaling Tue, 23 Jun 2026 00:00:00 GMT Poland's largest e-commerce site went down at the moment a campaign was driving peak traffic. AWS — The autoscaler that fought the network https://www.systemsfailed.dev/incident/aws-2021-autoscale https://www.systemsfailed.dev/incident/aws-2021-autoscale Tue, 23 Jun 2026 00:00:00 GMT A major us-east-1 disruption affecting a long list of AWS services and the customers built on them. AWS — A typo took out us-east-1 https://www.systemsfailed.dev/incident/aws-s3-2017-typo https://www.systemsfailed.dev/incident/aws-s3-2017-typo Tue, 23 Jun 2026 00:00:00 GMT S3 in us-east-1 down for hours; thousands of dependent sites and services degraded — including AWS's own status dashboard. Chef — Health checks too impatient to live https://www.systemsfailed.dev/incident/chef-2014-healthcheck https://www.systemsfailed.dev/incident/chef-2014-healthcheck Tue, 23 Jun 2026 00:00:00 GMT The Supermarket community site crashed two hours after launch with intermittent unresponsiveness. CircleCI — When GitHub came back, the queue fell over https://www.systemsfailed.dev/incident/circleci-github-recovery https://www.systemsfailed.dev/incident/circleci-github-recovery Tue, 23 Jun 2026 00:00:00 GMT Build throughput collapsed to roughly one transaction per minute; the backlog couldn't drain. Cloudflare — The regex that ate every CPU https://www.systemsfailed.dev/incident/cloudflare-2019-regex https://www.systemsfailed.dev/incident/cloudflare-2019-regex Tue, 23 Jun 2026 00:00:00 GMT Global 502s across ~10% of the web — Discord, Shopify, and Cloudflare's own dashboard went dark. Cloudflare — A backbone change crushed 19 data centers https://www.systemsfailed.dev/incident/cloudflare-2022-backbone https://www.systemsfailed.dev/incident/cloudflare-2022-backbone Tue, 23 Jun 2026 00:00:00 GMT 19 of Cloudflare's busiest data centers dropped offline at once, hitting a large share of global traffic. Discord — A cloud live-migration tipped over Redis https://www.systemsfailed.dev/incident/discord-redis-failover https://www.systemsfailed.dev/incident/discord-redis-failover Tue, 23 Jun 2026 00:00:00 GMT A real-time outage that forced a full service restart, reconnecting millions of clients over ~20 minutes. Fastly — One customer's config, the whole CDN down https://www.systemsfailed.dev/incident/fastly-2021-config https://www.systemsfailed.dev/incident/fastly-2021-config Tue, 23 Jun 2026 00:00:00 GMT 85% of Fastly's network returned errors within a minute — gov.uk, Reddit, Amazon and major news sites went down together. GitHub — 43 seconds that cost 24 hours https://www.systemsfailed.dev/incident/github-2018-splitbrain https://www.systemsfailed.dev/incident/github-2018-splitbrain Tue, 23 Jun 2026 00:00:00 GMT A day of stale, read-only metadata; webhooks and Pages builds frozen; manual write reconciliation afterward. GitLab — rm -rf on the wrong database https://www.systemsfailed.dev/incident/gitlab-2017-rmrf https://www.systemsfailed.dev/incident/gitlab-2017-rmrf Tue, 23 Jun 2026 00:00:00 GMT ~300GB of production database gone; six hours of issues, merge requests and users lost for good. Knight Capital — $440M in 45 minutes https://www.systemsfailed.dev/incident/knight-2012-powerpeg https://www.systemsfailed.dev/incident/knight-2012-powerpeg Tue, 23 Jun 2026 00:00:00 GMT A runaway trading algorithm lost roughly $460M in 45 minutes and effectively ended the firm. Meta — The day Facebook deleted itself from the internet https://www.systemsfailed.dev/incident/meta-2021-bgp https://www.systemsfailed.dev/incident/meta-2021-bgp Tue, 23 Jun 2026 00:00:00 GMT Facebook, Instagram, WhatsApp gone globally for ~6 hours. Internal tools, badge access and conference systems were down too, slowing recovery. Roblox — 73 hours down via one coordination layer https://www.systemsfailed.dev/incident/roblox-2021-consul https://www.systemsfailed.dev/incident/roblox-2021-consul Tue, 23 Jun 2026 00:00:00 GMT A near three-day full platform outage over Halloween weekend. Slack — The first-Monday thundering herd https://www.systemsfailed.dev/incident/slack-2021-herd https://www.systemsfailed.dev/incident/slack-2021-herd Tue, 23 Jun 2026 00:00:00 GMT Widespread connection failures on the first work-Monday of the year, exactly when everyone logged on at once.