rm -rf on the wrong database

Jan 2017Dec 2017

~300GB of production database gone; six hours of issues, merge requests and users lost for good.

Lesson

Backups you've never restored are not backups. The deletion is the famous part; the broken recovery path is the real lesson.

Trigger

An engineer fighting spam-induced replication lag ran a directory removal against what turned out to be the primary, not the replica.

Mechanism

The data vanished in seconds. Then the gut-punch: five of five backup and replication methods were broken or untested. A six-hour-old LVM snapshot, taken by luck, was the only thing that saved most of the data.

Interview lens

Disaster recovery: define RPO/RTO and, above all, prove your restores actually work.

Recurring patterns