Config change killed Seoul region DNS for 84 minutes
EC2 instances in the AWS AP-NORTHEAST-2 (Seoul) region were unable to resolve DNS for 84 minutes, affecting all customers running workloads in that region.
Configuration parameters that control minimum capacity thresholds must have semantic validation — silently defaulting to zero when a floor setting is removed is a single config push away from a regional outage.
A configuration update to the EC2 DNS resolver fleet incorrectly removed the setting specifying the minimum number of healthy hosts, causing the system to fall back to a dangerously low default value.
With the minimum healthy hosts threshold effectively zeroed out, the fleet shed healthy hosts below the capacity needed to serve queries → DNS resolution from within EC2 instances began failing across the entire Seoul region; recovery required manually stopping further host removal and then restoring fleet capacity to prior levels.
How would you design a configuration management system for a critical DNS resolver fleet so that removing or zeroing a minimum-capacity setting requires explicit confirmation rather than silently adopting a dangerous default?
