Chef/2014-07-10/hours
Health checks too impatient to live
Jan 2014Dec 2014
The Supermarket community site crashed two hours after launch with intermittent unresponsiveness.
Lesson
Aggressive health checks turn a latency blip into an outage. Set timeouts comfortably above your real tail latency.
Trigger
Launch-day load caused intermittent latency spikes.
Mechanism
Health-check timeouts were set so low that a brief latency blip made healthy nodes look dead. They were pulled from rotation, concentrating load on the rest, causing more latency — a self-reinforcing removal spiral.
Interview lens
Why health-check thresholds must account for p99 latency, not just the mean.
Recurring patterns
