rm -rf on the wrong database

Jan 2017Dec 2017

~300GB of production database gone; six hours of issues, merge requests and users lost for good.

Lesson

Backups you've never restored are not backups. The deletion is the famous part; the broken recovery path is the real lesson.

Trigger

An engineer fighting spam-induced replication lag ran a directory removal against what turned out to be the primary, not the replica.

Mechanism

The data vanished in seconds. Then the gut-punch: five of five backup and replication methods were broken or untested. A six-hour-old LVM snapshot, taken by luck, was the only thing that saved most of the data.

Interview lens

Set acceptable recovery time and data loss, then prove your backups can meet both by restoring real data.

Recurring patterns