Unnecessary Slack transactions starved database connection pool
incident.io's API experienced intermittent 20-second timeouts over two weeks in early 2023, causing customers to see request failures whenever connection pool slots were fully occupied.
Instrument your database connection pool with per-operation duration metrics before an incident forces you to — connection pool exhaustion from many unnecessary short transactions is invisible without that data, and the root cause will look like a general slowdown with no obvious culprit.
Every submission of a Slack modal (the /inc slash command confirmation) was silently opening a database transaction for the full duration of the request, holding a connection pool slot even when no transactional guarantees were needed.
High volumes of concurrent Slack modal submissions each held a connection pool slot for their full duration → all available connections were consumed → other API requests queued waiting for a connection → requests hit the 20-second timeout and failed; the intermittent nature made root cause identification difficult across 24 separate fix attempts over two weeks.
Track which endpoints hold database connections and for how long, not only how full the pool is.
