During a network split, decide whether consistency or availability matters more, and account for data loss from delayed replication.
43 seconds that cost 24 hoursRead full incident reportInterview field guide
Build stronger answers from real failures.
Use concise lessons from public postmortems to explain tradeoffs, failure modes, and safeguards in a system design interview. Each incident appears under every topic it can help you discuss.
- Incidents
- 41
- Failure topics
- 10
Find a talking point
Browse by failure mode
Jump to a topic or search by technology, company, incident, or lesson.
All interview topics
Failure modeSplit-brain2 casesA partition leaves two nodes both acting as primary — writes diverge.
1 / 2
Failure modeCascading failure18 casesOne component's failure overloads its neighbors until the system folds.
Renew certificates automatically and alert before they expire. Slow reconnection attempts so one expiry does not overload shared APIs.
Expired internal SSL cert breaks Actions runner connectivityRead full incident report1 / 18
Failure modeThundering herd9 casesA synchronized surge — retries, reconnects, cache stampede — buries a resource.
Give public and signed-in traffic separate capacity and limits, so abuse cannot starve signed-in users.
Public endpoints overwhelmed by abusive trafficRead full incident report1 / 9
Failure modeConfig change10 casesA configuration or rule push, not a code bug, takes production down.
A successful request does not mean the model answered well. Test every serving path and record the route without storing user content.
Claude requests routed to the wrong context serversRead full incident report1 / 10
Failure modeResource exhaustion12 casesCPU, memory, connections, file descriptors or disk run dry.
Scale on every constrained layer, not just the application process. Cap retries with budgets and backoff so partial failures do not multiply load.
Sidecar autoscaling failure cascades across GitHubRead full incident report1 / 12
Failure modeBad deploy11 casesA rollout, flag flip, or migration that wasn't safely staged.
A successful request does not mean the model answered well. Test every serving path and record the route without storing user content.
Claude requests routed to the wrong context serversRead full incident report1 / 11
Failure modeData loss1 caseAccidental deletion or replica divergence destroys or strands data.
Set acceptable recovery time and data loss, then prove your backups can meet both by restoring real data.
rm -rf on the wrong databaseRead full incident report Failure modeDNS / BGP4 casesName resolution or route withdrawal makes systems unreachable.
Keep each site's DNS independent from database maintenance, and test that failover still preserves connectivity between sites.
Database Migration Cascades Into Site-Wide DNS OutageRead full incident report1 / 4
Failure modeDependency failure6 casesAn upstream or downstream service — internal or third-party — gives out.
Renew certificates automatically and alert before they expire. Slow reconnection attempts so one expiry does not overload shared APIs.
Expired internal SSL cert breaks Actions runner connectivityRead full incident report1 / 6
Failure modeAutomation misfire9 casesA failover, autoscaler, or cleanup job does the wrong thing, confidently.
Delayed health reports can make healthy hosts look dead. Limit how quickly automated recovery can move work between hosts.
Kinesis cell manager mistakes healthy hosts for deadRead full incident report1 / 9