$440M in 45 minutes

Jan 2012Dec 2012

A runaway trading algorithm lost roughly $460M in 45 minutes and effectively ended the firm.

Lesson

Incomplete deploys are landmines. Don't reuse flags, delete dead code, and keep a tested kill switch for anything that touches money.

Trigger

New SEC-mandated code was deployed to 8 servers — but one server didn't get the update. A repurposed feature flag reactivated 8-year-old dead code ("Power Peg") on that machine.

Mechanism

The revived code began firing millions of unintended orders into the market. With no kill switch and ambiguous alerting, it ran unchecked for ~45 minutes before anyone could stop it.

Interview lens

Deployment safety and idempotency: detect partial rollouts; never let one un-updated node run divergent logic.

Recurring patterns