Data loss

Accidental deletion or replica divergence destroys or strands data.

1 incident on record

GitLab2017-01-31/~18h + permanent loss

rm -rf on the wrong database

~300GB of production database gone; six hours of issues, merge requests and users lost for good.

Lesson

Backups you've never restored are not backups. The deletion is the famous part; the broken recovery path is the real lesson.