systemsfailed.devfailure archive
← All incidents

Failure class

Dependency failure

An upstream or downstream service — internal or third-party — gives out.

6 incidents on record

GitHub2026-07-19/4 h 50 min

Expired internal SSL cert breaks Actions runner connectivity

Self-hosted and larger GitHub Actions runners worldwide could not connect for nearly 5 hours, delaying or failing workflow jobs, while the resulting reconnection surge briefly degraded API Requests, Issues, and Pages with added latency and elevated error rates.

Lesson

Certificate expiration is a fully predictable failure mode, so renewal must be automated with independent expiry monitoring, and reconnect paths need circuit breakers so a recovering fleet of clients doesn't stampede shared APIs on its way back up.

incident.io2025-10-20/~10h 30min (07:11–17:37 UTC, multiple services)

AWS us-east-1 outage cascaded through four hidden third-party dependencies

incident.io's on-call notifications were delayed for over two hours, SAML authentication was intermittently unavailable, Scribe's AI note-taker was offline for ~5 hours, and the deployment pipeline was blocked — all driven by AWS us-east-1 dependency propagation through their telecom provider, auth provider, transcription provider, and Docker Hub.

Lesson

Third-party provider risk runs much deeper than your direct dependencies — audit not just which services you integrate with but which cloud provider each one runs on, and verify that your deployment pipeline itself has no implicit dependencies on the same failing infrastructure you need to fix.

AWS2023-06-13/3h 48m

Lambda Frontend Scaling Bug Breaks US-EAST-1

A latent bug in Lambda's Frontend scaling logic degraded function invocations in one cell of US-EAST-1 for nearly four hours, causing elevated errors and latency across STS, Management Console, EKS cluster provisioning, Connect, and EventBridge.

Lesson

Autoscaling systems can hide defects that only trigger past capacity levels never previously exercised in production, so scaling logic must be tested well beyond historical peak, not just up to it.

Basecamp2014-03-24/1 h 40 min (45 min fully down)

Extortion DDoS took Basecamp down for 45 minutes

Basecamp and its sibling services were fully unreachable for 45 minutes and intermittently degraded for nearly two hours, affecting all customers on a Monday morning.

Lesson

Your status page must be load-tested for outage-level traffic spikes, and incident communication must begin within five minutes of detection — not after internal triage is complete.