# Exercise 1: Build the Alerting for a Checkout Service

## Scenario

`checkout` (Kubernetes, Prometheus + Alertmanager + Grafana) handles ~300 req/s. It depends on `cart`, `pricing`, `payments` (which calls an external PSP) and PostgreSQL. It also consumes a `payment-results` queue (Kafka) and must apply results within 2 minutes.

SLOs agreed with the business:
- **Availability**: 99.9% of `POST /checkout` requests return non-5xx over 28 days.
- **Latency**: 99% of `POST /checkout` requests complete in < 800 ms over 28 days.
- **Freshness**: 99% of payment results applied within 2 minutes.

Available metrics:
```
http_requests_total{service, route, method, code}
http_request_duration_seconds_bucket{service, route, le}
kafka_consumergroup_lag_seconds{group="checkout", topic="payment-results"}
up{job="checkout"}
kube_deployment_status_replicas_available{deployment="checkout"}
```

## Deliverables

1. **Error budgets**: compute each SLO's budget in minutes (or requests) per 28 days.
2. **Burn-rate alerting rules** (Prometheus YAML) for availability: the 14.4×/1h+5m and 6×/6h+30m pages, and the 1×/3d ticket. Include `for:`, labels (`severity`, `team`), and annotations (`summary`, `runbook_url`).
3. **Latency SLO alert**: express "fraction of requests slower than 800 ms" from the histogram (hint: the `le="0.8"` bucket) and write the 14.4× page rule.
4. **Freshness alert** on the Kafka lag metric.
5. **Two non-SLO alerts** that are still worth paging for (e.g. no traffic at all during business hours, zero available replicas) — justify why the SLO alerts don't cover them.
6. **Alertmanager config**: route `severity=page` to PagerDuty and `severity=ticket` to a Jira webhook; group by `alertname, service`; an inhibition rule so that a `CheckoutDown` alert suppresses the burn-rate alerts for the same service.
7. **Runbook** (bullet points) for the 14.4× availability alert: first 3 checks, first 3 mitigations.
8. **Time to detect**: with a 15 s scrape, 30 s rule evaluation, `for: 2m` and `group_wait: 30s`, estimate how long after a total outage starts the page arrives.

## Requirements Checklist

- [ ] Budgets: 0.1% of 28 days ≈ 40.3 min; 1% latency budget ≈ 403 min; freshness 1%.
- [ ] Availability rules use ratios of `rate()` over both windows joined with `and`, thresholds 14.4×0.001 and 6×0.001 (page) and 1×0.001 over 3d/6h (ticket).
- [ ] Latency SLI computed as `1 - (rate(bucket{le="0.8"}) / rate(_count))` (or `+Inf` bucket) and compared to 14.4 × 0.01.
- [ ] Freshness alert on `kafka_consumergroup_lag_seconds > 120` with a `for:`.
- [ ] Extra alerts: absent/very low traffic (`absent()` or rate below a floor) and `kube_deployment_status_replicas_available == 0` / `up == 0` — explained (no requests → no errors → SLO alerts stay silent).
- [ ] Alertmanager routes, grouping and an inhibit rule with `equal: [service]`.
- [ ] Runbook includes recent deploys/rollback, dependency dashboards (payments/PSP, DB), and mitigations (rollback, feature flag off, scale, failover).
- [ ] TTD estimate ≈ 15 s + 30 s + 120 s + 30 s ≈ 3–3.5 minutes.