Expired internal SSL cert breaks Actions runner connectivity
Self-hosted and larger GitHub Actions runners worldwide could not connect for nearly 5 hours, delaying or failing workflow jobs, while the resulting reconnection surge briefly degraded API Requests, Issues, and Pages with added latency and elevated error rates.
Certificate expiration is a fully predictable failure mode, so renewal must be automated with independent expiry monitoring, and reconnect paths need circuit breakers so a recovering fleet of clients doesn't stampede shared APIs on its way back up.
An SSL certificate used by a subset of internal services responsible for runner connectivity expired because of a lapse in certificate lifecycle management.
The expired certificate blocked self-hosted and larger runners from establishing connections to GitHub, causing queued and failing workflow jobs; as runners repeatedly retried connecting, the surge in reconnection traffic added several seconds of latency and elevated 5xx errors on GitHub's APIs, which in turn produced knock-on degradation in Issues and Pages before engineers rotated the certificate and the backlog drained.
When asked to design a large fleet of self-hosted job runners, discuss automated certificate lifecycle management, independent expiry alerting, and backoff/circuit-breaking on reconnect storms so a single expired credential can't cascade into platform-wide API degradation.
