systemsfailed.devfailure archive
← All incidents
Incident reportsource verified
GitHub/2025-09-15/25 min

A feature flag blocked every Copilot request

Jan 2025Dec 2025

Most GitHub Copilot features were degraded for 25 minutes, from 17:55 to 18:20 UTC, as the global rate limiter returned HTTP 403 responses for every request affected by the invalid configuration.

Lesson

Treat feature flags as production deploys with a state space. Validate partial and mixed-version combinations, canary global control points, and stop on sudden authorization anomalies. GitHub added traffic-anomaly monitors and expanded rate-limit scaling tests for the missed edge case.

A flag intended to reduce rate limiting for a subset of users was partially deployed to Copilot's global limiter, creating a configuration combination that preproduction tests had missed.

An edge case interpreted the intermediate flag state as an invalid global configuration. Instead of relaxing limits for the intended cohort, the shared control point rejected 100% of affected requests. Reverting the flag removed the bad state and produced immediate recovery.

Activate only complete rate-limit configurations, and test each one on a small traffic slice before wider release.

Share this failurePost on XDownload card