A cloud live-migration tipped over Redis

Jan 2020Dec 2020

A real-time outage that forced a full service restart, reconnecting millions of clients over ~20 minutes.

Lesson

Your cloud provider's routine automation — like live migration — is an event your failover code must survive. Test failover against real triggers, not just clean kills.

Trigger

Google Cloud automatically live-migrated the VM hosting Discord's primary Redis node.

Mechanism

The node briefly dropped offline; the HA cluster rebalanced and hit known bugs in Discord's Redis-failover handling. The partial outage exposed latent issues elsewhere, cascading through the real-time system until a full restart was required.

Interview lens

Failover correctness under provider-initiated events; chaos-testing the real failure triggers.

Recurring patterns