Resource exhaustion

An automated placement or failover system validated against typical workloads can still misbehave on a skewed profile — test cluster-management automation against shard-count extremes, not just throughput extremes, and cap its reaction rate so a false health signal cannot trigger a redistribution storm.

Autoscaling systems can hide defects that only trigger past capacity levels never previously exercised in production, so scaling logic must be tested well beyond historical peak, not just up to it.

Instrument your database connection pool with per-operation duration metrics before an incident forces you to — connection pool exhaustion from many unnecessary short transactions is invisible without that data, and the root cause will look like a general slowdown with no obvious culprit.

After migrating to a new cloud platform, probe for hidden per-node networking behaviors (like Dataplane V2 CPU amplification on connection churn) before blaming application-layer connection pools — node-level packet drops can mimic dozens of unrelated app failures at once.

When one coordination layer underpins everything, its failure is total. Know the failure modes of your service-discovery/consensus substrate before you scale onto it.

Your process manager may silently cap OS resource limits like LimitNOFILE to a value orders of magnitude below what you configured — always check the actual ulimits live in production processes, not just the config files.

Config changes need the same staged rollout as code. "It's just a rule" is how a global outage starts.

When a hot, shared table nears a known integer key ceiling, treat the migration as urgent and monitor actual headroom, not just awareness that a fix is 'in progress.'

Investing in architecture to reduce a known failure mode isn't enough — also build and rehearse the operational tooling needed to intervene quickly when that failure mode recurs at larger scale.

Recovery is a traffic event. The herd that hits you when an upstream comes back can be worse than its going down.

Your status page must be load-tested for outage-level traffic spikes, and incident communication must begin within five minutes of detection — not after internal triage is complete.