systemsfailed.devfailure archive
← All incidents

Failure class

Config change

A configuration or rule push, not a code bug, takes production down.

10 incidents on record

Anthropic2025-08-05/44 days to final rollout

Claude requests routed to the wrong context servers

Degraded Sonnet 4 responses initially affected 0.8% of requests, reached 16% in the worst hour, and touched about 30% of active Claude Code users at least once.

Lesson

Treat model-serving equivalence as a production SLO. Run sensitive quality evaluations continuously on the real serving matrix, segment signals by model, hardware, route, and context configuration, and connect feedback spikes to recent infrastructure changes while preserving user privacy.

GitHub2025-09-15/25 min

A feature flag blocked every Copilot request

Most GitHub Copilot features were degraded for 25 minutes, from 17:55 to 18:20 UTC, as the global rate limiter returned HTTP 403 responses for every request affected by the invalid configuration.

Lesson

Treat feature flags as production deploys with a state space. Validate partial and mixed-version combinations, canary global control points, and stop on sudden authorization anomalies. GitHub added traffic-anomaly monitors and expanded rate-limit scaling tests for the missed edge case.

GitHub2024-08-14/36 min

Config change silenced database health checks for 36 minutes

All GitHub.com services were fully inaccessible for all users worldwide for 36 minutes on the evening of August 14, 2024.

Lesson

Database configuration changes need staged rollout and automated fast-rollback — a single bad config push that breaks health check responses can cause the routing layer to simultaneously drop all database hosts, producing a total outage with no code change involved.

GitHub2020-02-19/8h 14min total across 4 events (Feb 19–27)

ProxySQL file descriptor limit silently capped by system process manager

GitHub.com experienced degraded service for a combined 8 hours 14 minutes across four events in late February 2020, with stalled writes on the mysql1 cluster affecting all authentication and core services that depended on it.

Lesson

Your process manager may silently cap OS resource limits like LimitNOFILE to a value orders of magnitude below what you configured — always check the actual ulimits live in production processes, not just the config files.

AWS2018-11-22/1 h 24 min

Config change killed Seoul region DNS for 84 minutes

EC2 instances in the AWS AP-NORTHEAST-2 (Seoul) region were unable to resolve DNS for 84 minutes, affecting all customers running workloads in that region.

Lesson

Configuration parameters that control minimum capacity thresholds must have semantic validation — silently defaulting to zero when a floor setting is removed is a single config push away from a regional outage.