# Exercise 1: Diagnose From the First 60 Seconds

Users report the PetPal API (`api-03`, 4 vCPU, 16 GiB, EBS gp2 200 GiB) is "very slow since about 14:00". You run the first-60-seconds checklist:

```
$ uptime
 14:22:07 up 41 days,  load average: 18.42, 15.10, 6.33

$ vmstat 1 4
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2 15      0 412000  21000 9800000    0    0  2100  5900 1800 3100  6  3 12 79  0
 1 16      0 409000  21000 9801000    0    0  2300  6100 1750 3000  5  3 10 82  0
 2 14      0 405000  21000 9802000    0    0  2200  6000 1820 3150  6  2 11 81  0

$ iostat -xz 1 2   (second sample)
Device   r/s    w/s   rkB/s   wkB/s  r_await w_await aqu-sz  %util
nvme1n1  95.0  610.0  2200.0  6000.0  48.20  96.50   61.3   100.0

$ df -h /var/lib/petpal
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    200G  141G   59G  71% /var/lib/petpal

$ pidstat -d 1 1
UID  PID   kB_rd/s  kB_wr/s  Command
 0   2210    10.0   5400.0   rsync
1001  900  2150.0    500.0   node

$ systemctl list-timers --all | grep 14:00
Mon 2026-10-20 14:00:00 UTC  ...  petpal-backup.timer  petpal-backup.service

$ dmesg -T | tail -3
(nothing unusual)
```

## Deliverables

1. Walk through each output and state what it tells you (and what it rules out).
2. What is the root cause? Explain why load is 18 with CPU mostly idle.
3. Why does an EBS **gp2** volume of 200 GiB make this worse? (Compute its baseline IOPS and explain burst credits.)
4. Immediate mitigation (minutes) — at least two options with trade-offs.
5. Permanent fixes (days) — at least three.
6. Which metrics/alerts would have caught this before users noticed?

## Requirements Checklist

- [ ] Reads load trend (rising), `b` column (blocked tasks) and `wa` ≈ 80% → I/O bound, not CPU; no swapping; no OOM.
- [ ] iostat: 100% util, high await, deep queue → saturated volume; pidstat identifies `rsync` (backup started 14:00 by the timer) as the main writer, competing with the app's reads.
- [ ] Load includes D-state tasks waiting on I/O → high load with idle CPUs.
- [ ] gp2 baseline = 3 IOPS/GiB × 200 = 600 IOPS (burst to 3,000 until credits are exhausted) — matches ~700 IOPS observed.
- [ ] Mitigation: throttle/stop the backup (`ionice -c3`, `rsync --bwlimit`, `systemctl stop petpal-backup`), or temporarily shift traffic to other instances.
- [ ] Permanent: gp3 with provisioned IOPS/throughput, snapshot-based backups instead of rsync, schedule off-peak with `IOSchedulingClass=idle`/`IOWeight`, separate volumes for data and backups.
- [ ] Alerts: disk await/queue depth, EBS BurstBalance, iowait, p95 latency SLO.