Troubleshooting a Linux Server

1A Method, Not Random Commands

When a server is "slow" or "broken", work systematically:

1. Define the problem: what's slow/failing, since when, for whom, what changed (deploys, config, traffic, certificates expiring)?
2. Check the big four resources with the USE method — for each resource, look at Utilization, Saturation and Errors: CPU, memory, disk I/O, network (plus file descriptors, connections, threads).
3. Look at the application (logs, error rates, latency, dependencies).
4. Form a hypothesis, test it, change one thing at a time, and write down what you did (the timeline feeds the post-mortem — devops-cloud/observability/).

2The First 60 Seconds

uptime                      # load averages 1/5/15 min — trend up or down?
dmesg -T | tail -20         # kernel: OOM kills, disk errors, network drops
vmstat 1 5                  # r (run queue), b (blocked), si/so (swap), us/sy/wa/st CPU
mpstat -P ALL 1 3           # per-CPU: one core pegged = single-threaded bottleneck
pidstat 1 3                 # per-process CPU
iostat -xz 1 3              # per-disk: %util, await (latency), r/s w/s, aqu-sz
free -m                     # memory: look at "available", not "free"
sar -n DEV 1 3              # network throughput per interface
sar -n TCP,ETCP 1 3         # connection rate, retransmits
top / htop                  # who is consuming it

(This is Brendan Gregg's well-known checklist; sysstat provides mpstat/pidstat/iostat/sar.)

3Reading the Signals

Load average

Number of tasks running or waiting for CPU plus those in uninterruptible I/O wait (D). Compare with CPU count (nproc): load 16 on 16 cores may be fine; load 16 on 2 cores with low CPU usage points at I/O.

CPU

• High us (user): application work — profile it (perf top, language profilers, py-spy, node --prof).
• High sy (system): syscalls, context switches, networking — perf, strace -c -p PID.
• High wa (iowait): CPUs idle waiting for disk.
• High st (steal): the hypervisor gives your VM less CPU — noisy neighbours or exhausted burst credits (AWS T instances).

Memory

• Linux uses free memory for page cache; "available" is what matters.
• Swapping (si/so > 0 continuously) → severe slowdowns.
• OOM killer messages in dmesg/journalctl -k; containers: OOMKilled, exit code 137.
• Leaks: RSS of one process grows steadily (ps -o rss, pmap -x PID, app heap dumps).

Disk

• df -h / df -i — full disk or inodes (02-filesystem_and_permissions/).
• iostat -x: %util near 100% and high await → saturated; cloud volumes have IOPS/throughput limits and burst credits (EBS gp2!).
• iotop / pidstat -d — who's doing the I/O.

Network

• Retransmits, drops (ip -s link, nstat), connection states (ss -s, CLOSE-WAIT leaks), DNS latency (03-networking_tools/).
• Limits: conntrack table full (dmesg: nf_conntrack: table full), ephemeral port exhaustion, accept queue overflows.

4Inspecting a Specific Process

systemctl status app ; journalctl -u app --since "30 min ago"
ls /proc/PID/fd | wc -l ; cat /proc/PID/limits      # fd usage vs limit
strace -f -tt -p PID -e trace=network,file          # what syscalls is it stuck on?
strace -c -p PID                                     # syscall time summary (Ctrl-C)
lsof -p PID                                          # open files and sockets
cat /proc/PID/status | grep -E 'State|Threads|VmRSS'
gdb -p PID / py-spy dump --pid PID / jstack PID      # thread stacks
perf top -p PID                                      # hottest functions

strace adds heavy overhead — use briefly in production; eBPF tools (bcc/bpftrace: execsnoop, opensnoop, biolatency, tcpretrans) are low-overhead alternatives.

5Common Failure Patterns

Under the HoodWhere the Numbers Come From▾

Nearly every tool reads kernel counters from /proc and /sys: uptime and top read /proc/loadavg (an exponentially damped moving average of runnable + uninterruptible tasks, updated every 5 seconds); vmstat/mpstat read cumulative CPU jiffies from /proc/stat and compute deltas per interval; free parses /proc/meminfo (where MemAvailable is the kernel's estimate of memory available without swapping); iostat reads /proc/diskstats (time spent doing I/O → %util, total wait time / completed I/Os → await); ss queries sockets via netlink; ps walks /proc/[pid]/stat. That's why tools that sample deltas need an interval (the first line of vmstat is the average since boot — ignore it), why %util is misleading for SSDs/NVMe that serve many requests in parallel, and why container metrics differ from host metrics (cgroup files like memory.current and cpu.stat give the per-container view).

⚠️ Common Pitfalls

1. Restarting before collecting evidence — the problem disappears and returns.
2. Reading free as "out of memory" because of page cache.
3. Treating load average as CPU usage.
4. Ignoring dmesg (OOM, disk errors, conntrack).
5. Changing several things at once.
6. Forgetting cloud limits — burst credits, IOPS caps, instance network limits.

📋 Quick Reference

60 s: uptime · dmesg -T | tail · vmstat 1 · mpstat -P ALL 1 · pidstat 1 · iostat -xz 1 · free -m · sar -n DEV,TCP,ETCP 1 · top
USE: utilization / saturation / errors for CPU, memory, disk, network, fds
process: journalctl -u · /proc/PID/{limits,fd,status} · strace -c/-p · lsof -p · perf top · py-spy/jstack
patterns: load≠CPU · OOM 137 · df -i · lsof +L1 · CLOSE-WAIT · st steal · burst credits

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: linux_troubleshooting_a_server_theory.md · generated by tools/build_lessons.py