ProxySQL file descriptor limit silently capped by system process manager
GitHub.com experienced degraded service for a combined 8 hours 14 minutes across four events in late February 2020, with stalled writes on the mysql1 cluster affecting all authentication and core services that depended on it.
Your process manager may silently cap OS resource limits like LimitNOFILE to a value orders of magnitude below what you configured — always check the actual ulimits live in production processes, not just the config files.
An unexpectedly resource-intensive query was routed to the mysql1 master instead of the read replica pool, spiking load on ProxySQL beyond its effective connection limit — a limit far lower than the team realized.
Four distinct events, each with a different proximate trigger, all converged on the same bottleneck: ProxySQL breaking under connection load because its file descriptor limit was silently reduced from 1,073,741,824 to 65,536 by the system process manager. Event 1 (Feb 19): resource-heavy query misrouted to master overloaded ProxySQL. Event 2 (Feb 20): planned master promotion caused unexpected load spike. Event 3 (Feb 25): connections crossed the 65,536 ceiling and the silent capping was finally discovered. Event 4 (Feb 27): application query changes spiked master load again before the fix was fully in place.
You find ProxySQL failing under load but your config sets LimitNOFILE to 1 billion — where would you look to find what the actual running file descriptor limit is, and how would you reliably verify it won't silently change after a restart?
