# Troubleshooting a Linux Server

## 1. A Method, Not Random Commands

When a server is "slow" or "broken", work systematically:

1. **Define the problem**: what's slow/failing, since when, for whom, what changed (deploys, config, traffic, certificates expiring)?
2. **Check the big four resources** with the **USE method** — for each resource, look at **U**tilization, **S**aturation and **E**rrors: CPU, memory, disk I/O, network (plus file descriptors, connections, threads).
3. **Look at the application** (logs, error rates, latency, dependencies).
4. **Form a hypothesis, test it, change one thing at a time**, and write down what you did (the timeline feeds the post-mortem — `devops-cloud/observability/`).

## 2. The First 60 Seconds

```bash
uptime                      # load averages 1/5/15 min — trend up or down?
dmesg -T | tail -20         # kernel: OOM kills, disk errors, network drops
vmstat 1 5                  # r (run queue), b (blocked), si/so (swap), us/sy/wa/st CPU
mpstat -P ALL 1 3           # per-CPU: one core pegged = single-threaded bottleneck
pidstat 1 3                 # per-process CPU
iostat -xz 1 3              # per-disk: %util, await (latency), r/s w/s, aqu-sz
free -m                     # memory: look at "available", not "free"
sar -n DEV 1 3              # network throughput per interface
sar -n TCP,ETCP 1 3         # connection rate, retransmits
top / htop                  # who is consuming it
```

(This is Brendan Gregg's well-known checklist; `sysstat` provides `mpstat/pidstat/iostat/sar`.)

## 3. Reading the Signals

### Load average
Number of tasks **running or waiting** for CPU **plus** those in uninterruptible I/O wait (`D`). Compare with CPU count (`nproc`): load 16 on 16 cores may be fine; load 16 on 2 cores with low CPU usage points at **I/O**.

### CPU
- High `us` (user): application work — profile it (`perf top`, language profilers, `py-spy`, `node --prof`).
- High `sy` (system): syscalls, context switches, networking — `perf`, `strace -c -p PID`.
- High `wa` (iowait): CPUs idle waiting for disk.
- High `st` (steal): the hypervisor gives your VM less CPU — noisy neighbours or exhausted burst credits (AWS T instances).

### Memory
- Linux uses free memory for **page cache**; "available" is what matters.
- Swapping (`si`/`so` > 0 continuously) → severe slowdowns.
- **OOM killer** messages in `dmesg`/`journalctl -k`; containers: `OOMKilled`, exit code 137.
- Leaks: RSS of one process grows steadily (`ps -o rss`, `pmap -x PID`, app heap dumps).

### Disk
- `df -h` / `df -i` — full disk or inodes (`02-filesystem_and_permissions/`).
- `iostat -x`: `%util` near 100% and high `await` → saturated; cloud volumes have IOPS/throughput limits and burst credits (EBS gp2!).
- `iotop` / `pidstat -d` — who's doing the I/O.

### Network
- Retransmits, drops (`ip -s link`, `nstat`), connection states (`ss -s`, CLOSE-WAIT leaks), DNS latency (`03-networking_tools/`).
- Limits: conntrack table full (`dmesg: nf_conntrack: table full`), ephemeral port exhaustion, accept queue overflows.

## 4. Inspecting a Specific Process

```bash
systemctl status app ; journalctl -u app --since "30 min ago"
ls /proc/PID/fd | wc -l ; cat /proc/PID/limits      # fd usage vs limit
strace -f -tt -p PID -e trace=network,file          # what syscalls is it stuck on?
strace -c -p PID                                     # syscall time summary (Ctrl-C)
lsof -p PID                                          # open files and sockets
cat /proc/PID/status | grep -E 'State|Threads|VmRSS'
gdb -p PID / py-spy dump --pid PID / jstack PID      # thread stacks
perf top -p PID                                      # hottest functions
```

`strace` adds heavy overhead — use briefly in production; eBPF tools (`bcc`/`bpftrace`: `execsnoop`, `opensnoop`, `biolatency`, `tcpretrans`) are low-overhead alternatives.

## 5. Common Failure Patterns

| Symptom | Likely causes | Confirm with |
|---|---|---|
| high load, low CPU | disk I/O wait, NFS hang | `vmstat` b/wa, `iostat -x`, `ps` D state |
| app killed randomly | OOM killer | `dmesg -T | grep -i oom` |
| "No space left on device" | disk full, inodes full, deleted-open files | `df -h`, `df -i`, `lsof +L1` |
| "Too many open files" | fd limit / fd leak | `/proc/PID/limits`, `ls /proc/PID/fd | wc -l` |
| slow only sometimes | GC pauses, CPU steal, burst credits, cron jobs | `st` in vmstat, app GC logs, `list-timers` |
| connections fail under load | accept queue, conntrack, ports, pool limits | `ss -lnt` Recv-Q, `dmesg`, `ss -s` |
| everything slow after deploy | regression, config change, cache cold | deploy timeline, compare metrics, roll back |

## Under the Hood: Where the Numbers Come From

Nearly every tool reads kernel counters from **/proc** and **/sys**: `uptime` and `top` read `/proc/loadavg` (an exponentially damped moving average of runnable + uninterruptible tasks, updated every 5 seconds); `vmstat`/`mpstat` read cumulative CPU jiffies from `/proc/stat` and compute deltas per interval; `free` parses `/proc/meminfo` (where `MemAvailable` is the kernel's estimate of memory available without swapping); `iostat` reads `/proc/diskstats` (time spent doing I/O → `%util`, total wait time / completed I/Os → `await`); `ss` queries sockets via netlink; `ps` walks `/proc/[pid]/stat`. That's why tools that sample deltas need an interval (the first line of `vmstat` is the average since boot — ignore it), why `%util` is misleading for SSDs/NVMe that serve many requests in parallel, and why container metrics differ from host metrics (cgroup files like `memory.current` and `cpu.stat` give the per-container view).

![A troubleshooting flow: define the problem and recent changes, run the first-60-seconds checks, apply the USE method to CPU, memory, disk and network with the tools that reveal utilization, saturation and errors, then drill into the specific process with logs, strace, lsof and profilers, change one thing and record the timeline](diagrams/01-use-method.svg)

## Common Pitfalls

1. **Restarting before collecting evidence** — the problem disappears and returns.
2. **Reading `free` as "out of memory"** because of page cache.
3. **Treating load average as CPU usage**.
4. **Ignoring `dmesg`** (OOM, disk errors, conntrack).
5. **Changing several things at once**.
6. **Forgetting cloud limits** — burst credits, IOPS caps, instance network limits.

## Quick Reference

```
60 s: uptime · dmesg -T | tail · vmstat 1 · mpstat -P ALL 1 · pidstat 1 · iostat -xz 1 · free -m · sar -n DEV,TCP,ETCP 1 · top
USE: utilization / saturation / errors for CPU, memory, disk, network, fds
process: journalctl -u · /proc/PID/{limits,fd,status} · strace -c/-p · lsof -p · perf top · py-spy/jstack
patterns: load≠CPU · OOM 137 · df -i · lsof +L1 · CLOSE-WAIT · st steal · burst credits