Bad deploy

Treat model-serving equivalence as a production SLO. Run sensitive quality evaluations continuously on the real serving matrix, segment signals by model, hardware, route, and context configuration, and connect feedback spikes to recent infrastructure changes while preserving user privacy.

Treat feature flags as production deploys with a state space. Validate partial and mixed-version combinations, canary global control points, and stop on sudden authorization anomalies. GitHub added traffic-anomaly monitors and expanded rate-limit scaling tests for the missed edge case.

Treat model behavior as a launch-blocking production property: define explicit failure-mode evals, formally weigh qualitative warnings against engagement metrics, add an opt-in alpha, and keep a rehearsed rollback path. OpenAI first mitigated with a system-prompt update, then completed rollback in about 24 hours.

A quick mitigation for a localized infra failure can silently break cross-site connectivity elsewhere, so any DNS or routing fix should be validated for its effect on the broader topology before being treated as the resolution.

Evaluate retrieval and generation separately, measure source quality when evidence is sparse, and abstain when the query or evidence is unreliable. Google added nonsense detection, limited satire and user-generated content, restricted weak query classes, retained hard-news restrictions, and strengthened health protections.

Network and routing changes deserve the same staged rollout and review as application code. The "resilience project" is itself a risk window.

A rollback is not automatically safe when a schema or format change has already been written to storage in the new shape — verify backward AND forward read compatibility before treating rollback as the safe default.

A latent bug can sit quiet until ordinary input you don't control wakes it. That input is part of your test matrix.

Never expose raw Postgres sequences as meaningful user-facing identifiers — sequences do not guarantee gap-free increments and any failover, crash, or upgrade can silently advance them by up to 32; use an explicit MAX+1 approach instead.

Deployment systems that regenerate critical infrastructure config (like DNS zone files) must validate API responses for completeness before applying them — and must never depend on the very service they are rebuilding.

Incomplete deploys are landmines. Don't reuse flags, delete dead code, and keep a tested kill switch for anything that touches money.