# Exercise 2: Post-Mortem of a Metastable Failure

## The Incident

```
09:00  traffic normal: 3,000 req/s to orders-api; DB p99 40 ms; pool 40 per instance × 30 instances
09:12  a slow analytics query (missing index) runs every minute on the primary; DB p99 → 400 ms
09:13  orders-api client timeout to DB = 300 ms; timed-out queries are retried twice (3 attempts)
09:14  DB CPU 100%; queries queue; p99 → 2 s; most requests now time out and retry
09:15  autoscaling adds 30 orders-api instances (CPU-based) → 60 instances × 40 connections
09:16  DB hits max_connections (2,000); new instances crash-loop on startup migrations check
09:20  on-call kills the analytics job; DB load from analytics disappears
09:20–09:55  system stays down: DB still at 100% CPU, almost every request still times out
09:55  on-call scales orders-api down to 10 instances and disables retries via a flag → recovers in 3 minutes
```

## Deliverables

1. **Timeline analysis**: identify the **trigger** and the **sustaining effect(s)**. Why didn't removing the trigger at 09:20 restore service? (Define "metastable failure" in your own words.)
2. For each of these, explain its contribution: timeout < DB p99, 3 attempts per query, CPU-based autoscaling, connection pool × instance count, startup checks against the DB.
3. **Fixes**, grouped as: prevent the trigger, limit amplification, enable recovery. For each, name the pattern from `adv_resilience_and_reliability_patterns_theory.md`.
4. **Load shedding design** for orders-api: what signal (in-flight requests, queue time, DB pool wait time), what threshold, what's shed first, and what clients receive.
5. **Recovery runbook**: the steps on-call should take next time (in order), and which of them should be automated.
6. **Prevention**: one chaos experiment and one CI/CD guardrail that would have caught a piece of this before production.

## Requirements Checklist

- [ ] Trigger: the slow analytics query; sustaining effects: retries (3× load), timeouts shorter than the degraded latency (work wasted on requests that were already abandoned), and scale-out adding connections/load to the bottleneck.
- [ ] Metastable failure explained: a self-sustaining overloaded state that persists after the trigger is removed because the system's reaction (retries, timeouts, scaling) keeps load above capacity.
- [ ] Fixes include: indexing/isolating analytics (read replica), retry budgets or no DB-level retries, deadline propagation, DB connection limits aligned with instance counts (RDS Proxy / pooler), scaling on the right signal with a max tied to DB capacity, circuit breaker/load shedding at the API edge.
- [ ] Load shedding uses an early, cheap signal (concurrency limit or pool wait time) and returns 503 with `Retry-After`, prioritizing critical endpoints.
- [ ] Runbook: stop retries / shed load first, reduce concurrency to below DB capacity, then remove the trigger, then gradually restore; automate retry-disable and concurrency limits.
- [ ] Prevention: chaos test injecting DB latency under load; CI check for queries without indexes / migration lint.