Processes and systemd

1Processes

A process is a running program with its own PID, memory space, open file descriptors, environment, user/group IDs and a parent (PPID). Every process except PID 1 is created by fork() (copy the parent) + exec() (replace the image with a new program). Threads are processes that share memory (in Linux, both are "tasks").

ps aux --sort=-%mem | head            # all processes, biggest memory first
ps -eo pid,ppid,user,stat,%cpu,%mem,etime,cmd --forest   # tree with chosen columns
pgrep -af node                        # find by name
top / htop                            # live view (press P/M to sort, 1 for per-CPU)
pstree -p 1                           # tree from PID 1
cat /proc/1234/status                 # kernel's view: state, memory, threads, capabilities
ls -l /proc/1234/fd                   # open files and sockets

Process states (STAT column)

Zombies use no memory, only a PID slot; fix the parent (or kill it so PID 1 adopts and reaps the children). Many D processes → I/O problems; load average counts both R and D (05-troubleshooting_a_server/).

2Signals

kill 1234            # SIGTERM
kill -HUP 1234       # reload
kill -9 1234         # only if SIGTERM didn't work after a grace period
pkill -f "node server.js"

Exit codes: 0 success; 1–125 app errors; 126 not executable; 127 command not found; 128+N killed by signal N (137 = SIGKILL, often the OOM killer; 143 = SIGTERM).

kill -9 $(pgrep -f server.js)
# no cleanup: in-flight requests dropped,
# temp files left, DB transactions aborted

kill -TERM 1234        # app drains and exits
sleep 30
kill -0 1234 2>/dev/null && kill -KILL 1234   # only if still alive

3Priorities, Limits and Resources

• nice (−20…19) adjusts CPU priority: nice -n 10 ./batch.sh, renice +5 -p 1234; ionice -c3 for idle I/O priority.
• ulimit / LimitNOFILE cap per-process resources (open files, processes). "Too many open files" → raise the limit and look for fd leaks (ls /proc/PID/fd | wc -l).
• cgroups (control groups) limit and account CPU, memory and I/O for groups of processes — the foundation of containers and of systemd's resource control. The kernel OOM killer kills processes when memory runs out (see dmesg -T | grep -i oom).

4systemd

systemd is PID 1 on almost every modern distribution: it boots the system, starts and supervises services, manages logging (journald), timers, mounts, sockets and cgroups.

# /etc/systemd/system/petpal-api.service
[Unit]
Description=PetPal API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=petpal
Group=petpal
WorkingDirectory=/opt/petpal/api
EnvironmentFile=/etc/petpal/api.env
ExecStart=/usr/bin/node dist/server.js
Restart=on-failure
RestartSec=5
TimeoutStopSec=30           # SIGTERM, then SIGKILL after 30 s
LimitNOFILE=65536
MemoryMax=1G                # cgroup memory limit
CPUQuota=200%
# hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/petpal

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload                # after editing unit files
sudo systemctl enable --now petpal-api      # start now and at boot
systemctl status petpal-api                 # state, main PID, recent logs
sudo systemctl restart|reload|stop petpal-api
systemctl list-units --failed
systemctl cat petpal-api                    # effective unit incl. drop-ins
sudo systemctl edit petpal-api              # drop-in override (/etc/systemd/system/petpal-api.service.d/)
journalctl -u petpal-api -f                 # follow logs
journalctl -u petpal-api --since "1 hour ago" -p err
systemd-analyze security petpal-api         # hardening score

Unit types: .service, .timer (cron replacement), .socket (socket activation), .target (groups like multi-user.target), .mount, .path. Service types: simple (default; the process is the service), exec, forking (legacy daemons), oneshot (scripts), notify (app signals readiness via sd_notify).

Timers instead of cron

# petpal-backup.timer
[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true              # run missed jobs after downtime
RandomizedDelaySec=300
[Install]
WantedBy=timers.target

Timers give you logs in the journal, dependencies, resource limits and systemctl list-timers.

Under the HoodWhat systemctl start Does▾

systemctl sends a D-Bus request to PID 1. systemd loads the unit (plus drop-ins), resolves dependencies and ordering (Requires/Wants decide what starts; After/Before decide in which order), and enqueues a start job. For the service it creates a cgroup (/sys/fs/cgroup/system.slice/petpal-api.service) and applies its limits (MemoryMax, CPUQuota), forks a child, sets up the execution environment — user/group, working directory, environment, namespaces for PrivateTmp/ProtectSystem, capabilities, seccomp filters — and execs ExecStart. stdout/stderr are connected to journald, which indexes logs with metadata (unit, PID, priority). Because all of the service's processes live in its cgroup, systemd can track and stop every child, even double-forked ones: on stop it sends SIGTERM to the main process (or all, per KillMode), waits TimeoutStopSec, then SIGKILLs whatever remains. When the main process exits unexpectedly, systemd receives SIGCHLD, records the exit status and applies the Restart= policy (with StartLimitBurst preventing restart loops).

⚠️ Common Pitfalls

1. kill -9 first — skips cleanup, corrupts state.
2. Running services as root without hardening.
3. Forgetting daemon-reload after editing units.
4. Apps that ignore SIGTERM — every restart waits the full timeout then gets killed.
5. Confusing After= with Requires=/Wants= — ordering vs dependency.
6. Cron jobs without logs or locking — use timers.

📋 Quick Reference

ps aux --sort=-%mem · ps -eo pid,ppid,stat,cmd --forest · top/htop · /proc/PID/{status,fd}
states R S D T Z · signals TERM(15) KILL(9) HUP(1) INT(2) · exit 128+N (137 OOM/KILL, 143 TERM)
nice/renice/ionice · ulimit/LimitNOFILE · cgroups · dmesg | grep -i oom
systemctl daemon-reload · enable --now · status · restart · edit · list-units --failed · list-timers
journalctl -u svc -f --since -p err · systemd-analyze security svc

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: linux_processes_and_systemd_theory.md · generated by tools/build_lessons.py