Debugging a Redis cache stampede on Kubernetes over SSH at 2am

The page came in at 2am: latency spiking, database CPU pinned. The accounting website's cache was in front of an expensive per-route permission calculation, and something had gone wrong with it.

Into the pod

SSH to the node, exec into the pod, and straight into redis-cli. The pattern was textbook: a hot key had expired, and every request that missed it was recomputing the same result at once and hammering the database. A classic cache stampede.

The fix

Short term: staggered TTLs so the whole keyspace never expires in lockstep. Longer term: a lock around the recompute so only one request rebuilds a cold key while the others wait. The database CPU came back down and the pages stopped.