# Exercise 2: Post-Incident Review — "Autoscaling Didn't Save Us"

## The Incident (timeline, all times UTC)

```
19:58  TV ad airs; traffic rises from 1,500 to 7,000 req/s within 3 minutes
20:00  ASG target tracking (CPU 70%) triggers scale-out from 4 to 9 instances
20:01  ALB 5xx ratio 4%, p99 latency 6 s
20:04  new instances pass the EC2 status check and enter service; app needs 90 s more
       to warm its cache, during which /health returns 200 and requests time out
20:05  ASG adds 8 more instances (CPU still high on the cold ones) → 17 instances
20:06  RDS connections hit max_connections (17 × 40 = 680 > 600); new instances
       fail on DB connect, ALB marks them unhealthy, ASG replaces them (churn)
20:09  ASG reaches max-size 20; RDS CPU 100%
20:25  traffic falls; the system recovers on its own
20:40  scale-in removes 12 instances within 5 minutes; a smaller second spike at 20:45
       causes another 3 minutes of errors
```

Configuration: target tracking on average CPU 70%; health-check type **EC2**; health-check grace 30 s; default instance warmup unset; `/health` returns 200 as soon as the HTTP server starts; DB pool 40 per instance; no alarms other than CPU; no scheduled scaling though the ad schedule was known.

## Deliverables

1. **Root causes**: list at least five contributing causes, each mapped to a timeline entry and a section of `aws_aws_scalability_and_monitoring_theory.md` (or `09-security_groups`, `08-cloudwatch`, `05-rds`).
2. **Fixes**, each tied to a cause: scaling metric, health checks and readiness, warmup/grace, DB connection strategy, scale-in protection, scheduled/predictive scaling, alarms.
3. **Monitoring gaps**: which symptom alarms would have paged at 20:01, and which dashboard panels would have shown the DB connection exhaustion immediately?
4. **Verification plan**: a load test that reproduces the TV-ad curve and the pass criteria for the fixes.
5. Write a 5-line **summary for leadership** (impact, cause, fix, prevention) without jargon.

## Requirements Checklist

- [ ] Causes include: CPU target too high/lagging metric (use requests per target), no pre-scaling despite a known event, `/health` lies during warm-up (readiness), health-check type EC2 + short grace, missing instance warmup (double scale-out), DB connection pool × instance count exceeding `max_connections`, aggressive scale-in.
- [ ] Fixes: request-based target tracking with a lower target, scheduled scaling before ads, readiness endpoint that waits for warm-up, ELB health checks with grace ≥ warm-up, default instance warmup, RDS Proxy/smaller pools/max size aligned to DB capacity, slower scale-in (cooldown/ scale-in protection).
- [ ] Alarms on 5xx ratio and p99 latency (symptoms), plus DB connections and `UnHealthyHostCount` panels.
- [ ] Load test reproduces the ramp (1,500 → 7,000 req/s in 3 min) with explicit pass criteria (p99, error ratio, DB connections below limit).
- [ ] Leadership summary is plain-language and actionable.