# Exercise 2: Three Incidents, One Runbook

For each incident below, write the investigation as a **runbook entry**: symptoms → commands (in order, with what you expect to see) → root cause → fix → prevention.

### Incident A — "No space left on device", but `df -h` shows 35% free
The app fails to create session files in `/var/lib/petpal/sessions`. `df -h` shows 35% free on that filesystem.

### Incident B — disk full, but `du` can't find the files
`df -h /var` shows 100% used (50 GiB), but `du -xsh /var` reports only 18 GiB. Someone deleted a 30 GiB `app.log` an hour ago "to free space".

### Incident C — the API dies every few hours
`systemctl status petpal-api` shows `code=killed, signal=KILL` and restarts; app logs show nothing before the restart. The host has 8 GiB RAM; the Node.js process normally uses 600 MiB.

## Deliverables

For **each** incident:

1. The ordered list of commands you'd run and what output would confirm your hypothesis.
2. Root cause (be specific).
3. Immediate fix and the risk of the fix.
4. Prevention (monitoring, configuration, code changes).

Then:

5. Write a short **generic triage checklist** (≤ 12 lines) your on-call team can follow for "server is unhealthy" alerts, based on the USE method.

## Requirements Checklist

- [ ] A: `df -i` shows 100% inode usage; `find /var/lib/petpal/sessions -type f | wc -l` → millions of tiny files; fix: delete expired sessions (e.g. `find -mmin +1440 -delete`); prevention: TTL cleanup job/timer, store sessions in Redis, inode alerts.
- [ ] B: `lsof +L1` (or `lsof | grep deleted`) shows the process still holding the deleted log; fix: reload/restart the process or truncate `/proc/PID/fd/N`; prevention: logrotate with `postrotate` reload or `copytruncate`, log to journald/stdout, disk alerts.
- [ ] C: `dmesg -T | grep -i -E 'killed process|oom'` / `journalctl -k` shows the OOM killer; check memory growth (RSS over time, heap snapshots), `MemoryMax`, `--max-old-space-size`; fix leak, set limits so systemd restarts cleanly; alert on memory growth.
- [ ] Each runbook has commands in a sensible order with expected outputs and both immediate and preventive actions.
- [ ] Generic checklist covers: what changed, `uptime`/`dmesg`/`vmstat`/`iostat`/`free`/`ss -s`/`df -h -i`, service status and logs, escalate/rollback criteria.