During a network split, decide whether consistency or availability matters more, and account for data loss from delayed replication.
43 seconds that cost 24 hoursRead full incident reportInterview field guide
Build stronger answers from real failures.
Use concise lessons from public postmortems to explain tradeoffs, failure modes, and safeguards in a system design interview. Each incident appears under every topic it can help you discuss.
- Incidents
- 41
- Failure topics
- 10
Find a talking point
Browse by failure mode
Jump to a topic or search by technology, company, incident, or lesson.
All interview topics
Failure modeSplit-brain2 casesA partition leaves two nodes both acting as primary — writes diverge.
1 / 2
Failure modeCascading failure18 casesOne component's failure overloads its neighbors until the system folds.
Scale on every constrained layer, not just the application process. Cap retries with budgets and backoff so partial failures do not multiply load.
Sidecar autoscaling failure cascades across GitHubRead full incident report1 / 18
Failure modeThundering herd9 casesA synchronized surge — retries, reconnects, cache stampede — buries a resource.
Scale on every constrained layer, not just the application process. Cap retries with budgets and backoff so partial failures do not multiply load.
Sidecar autoscaling failure cascades across GitHubRead full incident report1 / 9
Failure modeConfig change10 casesA configuration or rule push, not a code bug, takes production down.
Activate only complete rate-limit configurations, and test each one on a small traffic slice before wider release.
A feature flag blocked every Copilot requestRead full incident report1 / 10
Failure modeResource exhaustion12 casesCPU, memory, connections, file descriptors or disk run dry.
Scale on every constrained layer, not just the application process. Cap retries with budgets and backoff so partial failures do not multiply load.
Sidecar autoscaling failure cascades across GitHubRead full incident report1 / 12
Failure modeBad deploy11 casesA rollout, flag flip, or migration that wasn't safely staged.
Activate only complete rate-limit configurations, and test each one on a small traffic slice before wider release.
A feature flag blocked every Copilot requestRead full incident report1 / 11
Failure modeData loss1 caseAccidental deletion or replica divergence destroys or strands data.
Set acceptable recovery time and data loss, then prove your backups can meet both by restoring real data.
rm -rf on the wrong databaseRead full incident report Failure modeDNS / BGP4 casesName resolution or route withdrawal makes systems unreachable.
Keep each site's DNS independent from database maintenance, and test that failover still preserves connectivity between sites.
Database Migration Cascades Into Site-Wide DNS OutageRead full incident report1 / 4
Failure modeDependency failure6 casesAn upstream or downstream service — internal or third-party — gives out.
Renew certificates automatically and alert before they expire. Slow reconnection attempts so one expiry does not overload shared APIs.
Expired internal SSL cert breaks Actions runner connectivityRead full incident report1 / 6
Failure modeAutomation misfire9 casesA failover, autoscaler, or cleanup job does the wrong thing, confidently.
Roll out database configuration gradually so one bad change cannot mark every host unhealthy at once.
Config change silenced database health checks for 36 minutesRead full incident report1 / 9