Backwards-incompatible schema change stalls all job execution
All job executors were blocked from running customer jobs for roughly 53 minutes, with elevated queue times continuing for machine executors for over an hour longer, affecting all CircleCI customers.
A rollback is not automatically safe when a schema or format change has already been written to storage in the new shape — verify backward AND forward read compatibility before treating rollback as the safe default.
A deploy changed a field type in the distribution service's database schema, and work written before the deploy became unreadable to the new code.
The schema change made pre-deploy data fail distribution, so the team rolled back — but the rollback then made post-deploy data unreadable instead, keeping errors flowing; a hotfix let the distributor ignore the field, and manual compute scaling to absorb the resulting thundering herd was slowed because new nodes provisioned too slowly, got marked unhealthy, and were terminated before they could help.
When asked about safe database migrations, discuss why a schema change must be readable by both old and new code during the rollout window, including during a rollback.
