systemsfailed.devfailure archive
← All incidents
Incident report
Basecamp/2018-11-09/~5 h

Primary key overflow on events table forces read-only mode

Jan 2018Dec 2018

Basecamp 3 was fully read-only for almost five hours, blocking all writes across the product — no new messages, todos, files, or edits — while reads continued to work.

Lesson

When a hot, shared table nears a known integer key ceiling, treat the migration as urgent and monitor actual headroom, not just awareness that a fix is 'in progress.'

The primary key on Basecamp's shared events table, which logs nearly every user action, hit the 32-bit integer ceiling of 2,147,483,647 and could no longer accept new rows.

A single high-traffic events table tracking nearly all product activity used a 32-bit integer primary key; the team was already mid-migration to widen it but ran out of headroom before finishing, and once the ceiling was hit every write-path action in the app failed simultaneously because it depended on writing to that table.

When asked about scaling a high-write table, discuss choosing 64-bit primary keys upfront and monitoring key-space headroom as a first-class production metric.

Share this failurePost on XDownload card