# Exercise 1: Investigate an Intermittent Checkout Failure

## The Report

> "Some customers get 'Something went wrong' at checkout. Started yesterday afternoon. Not everyone — maybe 3–5% of attempts. Support can't reproduce it."

## Evidence You Can Query (summarized)

**Metrics (last 48 h)**
- `POST /checkout` error rate: 0.2% until yesterday 14:10, then ~4%. Latency p50 unchanged; p99 from 600 ms to 3.1 s.
- Errors by region: eu-west 6.8%, us-east 0.3%.
- Errors by app version: web 0.4%; iOS 4.2.0 9.5%, iOS 4.1.x 0.3%; Android 0.5%.
- `payments` → PSP call latency p99 unchanged. `pricing` error rate unchanged.
- `orders-db` (eu-west replica) `ReplicaLag` spikes up to 12 s since yesterday 14:00.

**Change log**
- yesterday 13:55 — iOS 4.2.0 rolled out to 20% of users (new "express checkout" flow)
- yesterday 14:00 — eu-west database maintenance: read replica instance class changed
- yesterday 16:30 — `checkout` service deploy (logging changes only)

**Sample failed trace (eu-west, iOS 4.2.0)**
```
POST /checkout                      [0 → 3,050 ms]  status 500
├─ GET /cart/c-81 (cart)            [5 → 40]
├─ POST /orders (orders)            [45 → 120]   201 {orderId: o-9912}
├─ GET /orders/o-9912 (orders)      [125 → 160]  404   ← reads from replica
├─ (retry) GET /orders/o-9912       [1,160 → 1,190] 404
├─ (retry) GET /orders/o-9912       [3,000 → 3,030] 404
└─ error: "order o-9912 not found after create"
```

**Sample log line**
```json
{"level":"error","service":"checkout","trace_id":"4bf9…","app_version":"ios-4.2.0","region":"eu-west","msg":"order not found after create","order_id":"o-9912","attempt":3}
```

## Deliverables

1. **Scope** the problem: which dimensions concentrate the errors, and which don't? What does that rule out?
2. **Hypotheses**: list at least three candidate explanations consistent with the change log, then use the evidence to accept/reject each.
3. **Root cause**: write the causal chain (from the two changes to the 500) step by step. Explain why **both** iOS 4.2.0 *and* eu-west matter.
4. **Mitigation now** (minutes): at least two options with their trade-offs.
5. **Proper fix** (days): code/architecture changes, referencing read-your-writes (`databases/scaling/02-replication`) and idempotency.
6. **Observability gaps**: what would have made this faster to find (metrics, alerts, trace attributes)?
7. **Post-mortem summary**: impact, trigger, root cause, contributing factors, action items (owner + due date placeholders), written blamelessly.

## Requirements Checklist

- [ ] Scope: concentrated in eu-west and iOS 4.2.0 (express checkout); not PSP/pricing; the 16:30 deploy is ruled out by timing.
- [ ] Hypotheses weighed, including replica lag, the new iOS flow, and the logging deploy, with evidence for/against each.
- [ ] Root cause: the new express flow reads the order immediately after creating it, and that read goes to a lagging eu-west replica (lag increased after the maintenance) → 404 → retries within 3 s don't outlast up to 12 s of lag → 500.
- [ ] Mitigations: route post-create reads to the primary (flag/config), roll back or pause iOS 4.2.0 rollout / disable express checkout, fix replica performance; trade-offs discussed.
- [ ] Proper fix: return the created order from the write (no read-after-write), or read-your-writes routing (session pinned to primary / LSN check), plus replica-lag alerting.
- [ ] Observability: alert on replica lag, error-rate breakdown by app version/region, trace attribute for "read from replica vs primary".
- [ ] Post-mortem is blameless and lists concrete action items.