Kinesis cell manager mistakes healthy hosts for dead
For nearly seven hours in US-EAST-1, an internal Kinesis Data Streams cell degraded, raising latencies and error rates across CloudWatch Logs, S3 event delivery, Data Firehose, ECS, Lambda, Redshift, and Glue, with log backlogs taking until the next day to clear.
An automated placement or failover system validated against typical workloads can still misbehave on a skewed profile — test cluster-management automation against shard-count extremes, not just throughput extremes, and cap its reaction rate so a false health signal cannot trigger a redistribution storm.
A routine single-AZ deployment cycled hosts in and out of service on a recently migrated Kinesis cell whose workload — a huge number of very low-throughput shards — the new cell management system had never balanced before.
The cell manager balanced shards by throughput, so it packed the many low-throughput shards onto a few hosts; those hosts' periodic status messages grew so large they could not be processed in time, the manager falsely declared healthy hosts unhealthy and mass-redistributed their shards, and the redistribution spike overloaded the subsystem that provisions secure data-plane connections, impairing the whole cell.
Delayed health reports can make healthy hosts look dead. Limit how quickly automated recovery can move work between hosts.
