Automation misfire

An automated placement or failover system validated against typical workloads can still misbehave on a skewed profile — test cluster-management automation against shard-count extremes, not just throughput extremes, and cap its reaction rate so a false health signal cannot trigger a redistribution storm.

Database configuration changes need staged rollout and automated fast-rollback — a single bad config push that breaks health check responses can cause the routing layer to simultaneously drop all database hosts, producing a total outage with no code change involved.

When perception is uncertain after a collision, the minimum-risk fallback must prevent additional motion. Test rare, high-severity scenarios in simulation, constrain uncertain classifications from authorizing irreversible actuation, and evaluate the complete perception-to-action chain. Cruise's remedy would have kept the vehicle stationary.

Feedback loops between automated clients and the network they depend on cause congestion collapse. Monitoring must not share fate with the thing it monitors.

Never expose raw Postgres sequences as meaningful user-facing identifiers — sequences do not guarantee gap-free increments and any failover, crash, or upgrade can silently advance them by up to 32; use an explicit MAX+1 approach instead.

Your cloud provider's routine automation — like live migration — is an event your failover code must survive. Test failover against real triggers, not just clean kills.

Automated failover that reacts faster than your network heals can manufacture split-brain. Tune failover to application topology, not just node liveness.

Backups you've never restored are not backups. The deletion is the famous part; the broken recovery path is the real lesson.

Incomplete deploys are landmines. Don't reuse flags, delete dead code, and keep a tested kill switch for anything that touches money.