Unnecessary Slack transactions starved database connection pool
incident.io's API experienced intermittent 20-second timeouts over two weeks in early 2023, causing customers to see request failures whenever connection pool slots were fully occupied.
Instrument your database connection pool with per-operation duration metrics before an incident forces you to — connection pool exhaustion from many unnecessary short transactions is invisible without that data, and the root cause will look like a general slowdown with no obvious culprit.
Every submission of a Slack modal (the /inc slash command confirmation) was silently opening a database transaction for the full duration of the request, holding a connection pool slot even when no transactional guarantees were needed.
High volumes of concurrent Slack modal submissions each held a connection pool slot for their full duration → all available connections were consumed → other API requests queued waiting for a connection → requests hit the 20-second timeout and failed; the intermittent nature made root cause identification difficult across 24 separate fix attempts over two weeks.
How would you design observability for a shared database connection pool to identify which endpoints hold connections the longest and set meaningful exhaustion alerts?
