# Exercise 1: "The API Can't Reach the Database"

After a network change, `orders-api` on `app-01` (10.0.1.15) logs `connect ETIMEDOUT 10.0.2.30:5432` for ~30% of requests. The PostgreSQL server `db-01` (10.0.2.30) looks healthy. You have shell access to both hosts.

You collected:

```
app-01$ ip route
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.15
10.0.2.0/24 via 10.0.1.254 dev eth0

app-01$ nc -zv -w 3 10.0.2.30 5432     # run 10 times: 7 succeeded, 3 timed out
app-01$ ss -tan state syn-sent
SYN-SENT 0 1 10.0.1.15:51432 10.0.2.30:5432
SYN-SENT 0 1 10.0.1.15:51440 10.0.2.30:5432

db-01$ ss -lnt 'sport = :5432'
State  Recv-Q Send-Q Local Address:Port
LISTEN 0      200    0.0.0.0:5432

db-01$ sudo tcpdump -nn -i eth0 'tcp port 5432 and tcp[tcpflags] & tcp-syn != 0' -c 20
... 14 SYNs from 10.0.1.15 received, 14 SYN-ACKs sent ...

app-01$ sudo tcpdump -nn -i eth0 'tcp port 5432 and tcp[tcpflags] & tcp-syn != 0' -c 20
... 20 SYNs sent, 14 SYN-ACKs received ...
```

## Deliverables

1. Interpret each output: what it proves and what it rules out.
2. Where are the 6 missing SYNs being lost? Give your top two hypotheses (consider a second router/NAT path, asymmetric routing with a stateful firewall, conntrack table limits, a security group/NACL, ECMP to a broken next hop).
3. List the **next five commands** you'd run (on which host) to confirm, with the expected output for each hypothesis.
4. How would you tell "refused" from "timeout" from "DNS failure" in the app logs, and what does each imply?
5. What monitoring would have caught this before users did?

## Requirements Checklist

- [ ] Correctly states: Postgres listens on all interfaces with free accept queue (not an app/listen issue); SYNs leave app-01 but only 70% arrive at db-01 → loss on the path between them (not on db-01).
- [ ] Hypotheses include path-level issues (one of several next hops/NAT instances dropping, NACL/stateful firewall asymmetry, conntrack table full on a middlebox).
- [ ] Next commands are concrete (`traceroute -T -p 5432`, `mtr -T -P 5432`, tcpdump on the router 10.0.1.254, `dmesg | grep conntrack` on middleboxes, `ip route get 10.0.2.30`, cloud flow logs).
- [ ] Refused = RST (nothing listening/REJECT), timeout = drops, DNS = `ENOTFOUND`/`EAI_AGAIN`.
- [ ] Monitoring: synthetic TCP checks, connection error rate metrics by type, TCP retransmit metrics, flow logs alerts.