# Linux Networking Tools for DevOps

Network concepts (TCP, DNS, TLS, HTTP) are covered in `networking-web/fundamentals/`; this topic is the practical **toolbox** for inspecting and debugging them on a Linux server.

## 1. Interfaces, Addresses, Routes

```bash
ip -br addr                 # interfaces and IPs (brief)
ip link show                # state, MTU
ip route                    # routing table; "default via 10.0.0.1 dev eth0"
ip route get 8.8.8.8        # which interface/gateway would be used
ip neigh                    # ARP/neighbour table
cat /etc/resolv.conf        # DNS servers and search domains (systemd-resolved: resolvectl status)
hostname -I
```

## 2. Sockets and Ports: `ss`

`ss` (replacement for `netstat`) shows sockets:

```bash
ss -tlnp                    # TCP listening sockets, numeric, with process
ss -tunap                   # all TCP/UDP sockets with processes
ss -tan state established '( dport = :5432 )'   # connections to Postgres
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c   # count by state
ss -s                       # summary
```

| State | Meaning / smell |
|---|---|
| `LISTEN` | server waiting for connections (check the bind address: `127.0.0.1` = local only, `0.0.0.0`/`*` = all interfaces) |
| `ESTAB` | active connection |
| `TIME-WAIT` | closed by this side, waiting 2×MSL — many = lots of short-lived connections (use keep-alive/pooling) |
| `CLOSE-WAIT` | peer closed, **our app hasn't** — growing numbers = connection leak in the app |
| `SYN-SENT` | connecting — stuck = firewall/blackhole on the path |
| `SYN-RECV` | half-open on server — many = SYN flood or full accept queue |

`lsof -i :8080` shows which process holds a port.

## 3. Connectivity and Latency

```bash
ping -c 4 db.internal                  # ICMP reachability + RTT (may be blocked)
traceroute -T -p 443 api.example.com   # path with TCP probes (or mtr for continuous)
mtr -rwc 50 api.example.com            # per-hop loss/latency report
nc -zv db.internal 5432                # is the TCP port reachable? (-u for UDP)
timeout 3 bash -c '</dev/tcp/db.internal/5432' && echo open   # without nc
```

"Connection refused" = host reachable, nothing listening (or REJECT rule). "Timeout" = packets dropped (firewall, security group, routing, host down) — `networking-web/fundamentals/11-latency_and_troubleshooting`.

## 4. DNS

```bash
dig api.petpal.io +short
dig api.petpal.io A +noall +answer         # with TTLs
dig @8.8.8.8 api.petpal.io                 # ask a specific resolver
dig +trace petpal.io                       # follow delegation from the root
dig -x 203.0.113.10                        # reverse lookup
getent hosts api.petpal.io                 # what the system resolver (nsswitch, /etc/hosts) returns
```

`getent` vs `dig` difference reveals `/etc/hosts` overrides or search-domain effects (Kubernetes `ndots:5`!).

## 5. HTTP and TLS

```bash
curl -sv https://api.petpal.io/health -o /dev/null          # headers, TLS handshake, redirects
curl -s -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' https://api.petpal.io/
curl --resolve api.petpal.io:443:10.0.1.20 https://api.petpal.io/   # hit one backend, keep SNI/Host
openssl s_client -connect api.petpal.io:443 -servername api.petpal.io </dev/null | openssl x509 -noout -dates -subject -issuer
```

## 6. Packet Capture

```bash
sudo tcpdump -i any -nn port 5432 -c 50
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'   # SYNs and RSTs only
sudo tcpdump -i any -w /tmp/cap.pcap host 10.0.1.20                 # save for Wireshark
```

Capture on both ends to prove where packets disappear.

## 7. Firewalls

- **nftables** (modern) / **iptables** (legacy, often nft-backed): `sudo nft list ruleset`, `sudo iptables -L -n -v`.
- Front-ends: **ufw** (Ubuntu: `ufw allow 22/tcp`), **firewalld** (RHEL: `firewall-cmd --list-all`).
- In the cloud, security groups/NACLs act *before* the host firewall (`devops-cloud/aws/09-security_groups`); Kubernetes adds NetworkPolicies and kube-proxy rules (`devops-cloud/kubernetes/04-ingress_and_networking`).

## Under the Hood: The Path of an Incoming Connection

A SYN arrives at the NIC → the driver hands it to the kernel network stack → **netfilter** hooks (PREROUTING: DNAT, e.g. Docker/Kubernetes port mappings; INPUT: host firewall filtering) decide its fate → if a socket is **listening** on that address:port, the kernel replies SYN-ACK and holds the half-open connection in the **SYN queue**; on the final ACK it moves to the **accept queue** (sized by the `listen()` backlog and `net.core.somaxconn`) → the application's `accept()` takes it, getting a new socket with its own file descriptor. If nothing listens, the kernel sends RST ("connection refused"); if a firewall drops it, the client retransmits SYNs until timeout. If the app is too slow to `accept()`, the accept queue overflows and SYNs get dropped (visible in `nstat -az | grep -i listen` / `ss -lnt` Recv-Q). On close, the side that closes first enters TIME-WAIT; the other side sits in CLOSE-WAIT until its application calls `close()`.

![An incoming SYN passes the NIC and netfilter PREROUTING and INPUT chains; if a socket listens, the handshake moves the connection from the SYN queue to the accept queue until the app calls accept; no listener means RST and a firewall drop means timeouts; tools like ss, tcpdump, nft and curl inspect each stage](diagrams/01-connection-path.svg)

## Common Pitfalls

1. **Service bound to 127.0.0.1** while you test from another host.
2. **Treating "refused" and "timeout" as the same**.
3. **Testing with `ping`** when ICMP is blocked but TCP works (or vice versa).
4. **Ignoring CLOSE-WAIT growth** — an app-side leak.
5. **Forgetting cloud firewalls** (security groups) when the host firewall looks fine.
6. **DNS caching** — old answers until TTL expires.

## Quick Reference

```
ip -br addr · ip route (get) · resolvectl status
ss -tlnp · ss -tan state … · ss -s · lsof -i :PORT
ping · mtr · traceroute -T · nc -zv host port · </dev/tcp/host/port
dig +short / +trace / @server · getent hosts
curl -sv · curl -w timings · curl --resolve · openssl s_client
tcpdump -nn -i any port X (-w file) · nft list ruleset · ufw / firewalld