# Exercise 2: Replace a Noisy Alarm Set

The `orders` platform (ALB → ECS service → RDS PostgreSQL, plus an SQS queue consumed by a Lambda) currently has these alarms, which page on-call:

| # | Alarm | Condition |
|---|---|---|
| 1 | EC2/ECS CPU | > 70% for 1 minute |
| 2 | Any 5xx | `HTTPCode_Target_5XX_Count` ≥ 1 in 1 minute |
| 3 | Latency | `TargetResponseTime` Average > 300 ms for 1 minute |
| 4 | RDS CPU | > 60% for 5 minutes |
| 5 | SQS queue | `ApproximateNumberOfMessagesVisible` > 100 |
| 6 | Lambda | `Errors` ≥ 1 |
| 7 | Heartbeat | custom metric `Shop/Jobs NightlyExportCompleted` < 1 — missing data treated as `notBreaching` |

Last month: 212 pages, 9 real incidents. One real incident was missed: the nightly export stopped running for 4 days.

## Deliverables

1. For each alarm, decide **keep / change / delete / downgrade to ticket**, and give the new definition where applicable (metric or metric math, statistic, period, M-of-N, threshold, missing-data treatment).
2. Explain exactly why alarm 7 never fired.
3. Design a **composite alarm** that pages only when the user-facing symptom is real (e.g. high 5xx ratio AND/OR high p99 latency), and list which component alarms feed it.
4. Add an alarm for the SQS consumer that reflects **user impact** better than queue depth.
5. Propose a **Synthetics canary** for the checkout flow: what it does, how often, and what it alerts on.

## Requirements Checklist

- [ ] CPU alarms (1, 4) deleted or downgraded to tickets/scaling signals (causes, not symptoms).
- [ ] 5xx alarm changed to a **ratio** via metric math (e.g. > 1% for 3 of 5 minutes) with `notBreaching` missing data.
- [ ] Latency alarm uses **p99** (or p95) instead of Average, with M of N.
- [ ] Alarm 7 explained: when the job doesn't run, no datapoint is published, and missing data treated as `notBreaching` keeps it OK → must be `breaching` (or alarm on "missing").
- [ ] SQS alarm on `ApproximateAgeOfOldestMessage` (staleness users feel) instead of raw depth; Lambda errors as a ratio or via DLQ depth.
- [ ] Composite alarm built from the ratio and latency alarms (and optionally a canary failure).
- [ ] Synthetics canary scripted against the checkout journey every 1–5 minutes, alarming on consecutive failures.