# Debugging Distributed Systems

## 1. Why It's Different

In a single process you can set a breakpoint, step through, and see the whole state. In a distributed system:

- the "program" spans many processes, machines and teams;
- failures are **partial** (some instances, some regions, some requests);
- timing matters — bugs depend on interleavings, latency, retries, clock skew;
- you usually can't reproduce production conditions locally;
- the act of observing (adding logs, attaching debuggers) can change timing.

So debugging becomes **investigation from telemetry**, guided by hypotheses — closer to science than to stepping through code.

## 2. A Method

```
1. define the symptom precisely   what fails? for whom? since when? how much? (error rate, latency, affected %)
2. scope it                        which endpoints/regions/versions/tenants/instances? which not?
3. look for what changed           deploys, config/flag changes, traffic shape, dependencies, infra events
4. form a hypothesis               "requests from app v4.2 to /checkout time out on payments since 14:05"
5. test it with data               dashboards, traces, logs filtered by the hypothesis; compare good vs bad
6. mitigate first                  rollback, disable flag, shift traffic, scale — restore service before root-causing
7. find the root cause             5 whys / causal analysis on the preserved evidence
8. prevent recurrence              fix + tests + alert/runbook improvements (blameless post-mortem)
```

**Compare good with bad**: the fastest way to localize a problem is to split traffic by attributes (version, region, instance, customer, endpoint, payload size) and find the dimension where errors concentrate. High-cardinality, structured telemetry (`devops-cloud/observability/`) is what makes that possible.

## 3. The Tools and What They Answer

| Question | Tool |
|---|---|
| is something wrong, how bad, since when? | metrics / SLO dashboards, deploy markers |
| which requests? which hop is slow/failing? | distributed traces (`system-design/system-design/06-distributed_tracing`) |
| what exactly happened in this request? | logs filtered by trace ID |
| which code is hot / stuck? | continuous profiling, thread dumps, `async` stack traces |
| what is the system doing right now? | live queries: `pg_stat_activity`, queue depths, connection pools, `kubectl get events` |
| network-level issues? | `curl -w` timings, `dig`, `tcpdump`/Wireshark, `mtr` (`networking-web/fundamentals/11-latency_and_troubleshooting`) |
| can I reproduce it? | replaying recorded traffic in staging, fault injection, load tests |

## 4. Common Culprits (Check These Early)

1. **A recent change** — deploy, config, feature flag, dependency upgrade, infra change. (A large share of incidents.)
2. **Timeouts and retries** interacting (`06-…`, `07-…`) — amplification, metastable failures.
3. **Resource exhaustion** — connection pools, file descriptors, memory, thread pools, disk, rate-limit quotas.
4. **Data-dependent bugs** — one tenant's huge payload, one malformed record (poison message), unicode/timezones.
5. **Clock and ordering** — skew between hosts, events out of order, TTLs and token expiry.
6. **Caching** — stale or poisoned entries, cache stampedes.
7. **Partial deploys / version skew** — old and new versions talking with incompatible schemas.
8. **DNS and certificates** — expired certs, stale DNS caches.

## 5. Distributed-System Bug Patterns

- **Heisenbugs** — disappear when observed (timing). Use always-on structured telemetry rather than ad-hoc debug logging.
- **Split brain** — two nodes both think they're leader (check fencing tokens, quorum).
- **Lost updates** — concurrent writers (`02-race_conditions/`), check versions/CAS.
- **Duplicate processing** — at-least-once delivery without idempotency.
- **Thundering herds** — synchronized retries, cache expiry, cron jobs at :00.
- **Cascading failures** — one slow dependency saturating callers (`07-resilience_and_reliability_patterns/`).

## 6. Making Systems Debuggable (Before You Need It)

- Structured logs with trace IDs, service version, region, tenant (`devops-cloud/observability/01-logging`).
- RED metrics per endpoint and per dependency; saturation metrics for every pool and queue.
- Tracing with sampling that keeps errors and slow requests.
- Deploy/flag events as annotations on dashboards.
- Runbooks per alert; dashboards that start from the user symptom.
- Debug endpoints (protected): build info, config (redacted), pool stats, feature flags.
- The ability to **turn up logging for one tenant/request** at runtime without a deploy.

## Under the Hood: Why Clocks Make Causality Hard

Timestamps from different machines aren't directly comparable: NTP keeps clocks within milliseconds normally, but skew of tens or hundreds of ms happens, and a VM pause or leap-second smear can move clocks unexpectedly. Sorting a merged log by wall-clock time can therefore show an effect *before* its cause. Distributed systems establish **causal order** differently: Lamport timestamps (a counter incremented on every event and set to `max(local, received) + 1` on message receipt) give an order consistent with causality; vector clocks detect concurrency; and tracing's **parent/child span links** encode causality explicitly, which is why traces are more trustworthy than timestamp-sorted logs when reconstructing "what caused what". When comparing timings across hosts, prefer durations measured on one host (span durations) to differences between two hosts' timestamps.

![Debugging loop: define and scope the symptom, check recent changes, form a hypothesis, test it by comparing good and bad requests across metrics, traces and logs, mitigate first, then find the root cause and prevent recurrence](diagrams/01-debugging-loop.svg)

## Common Pitfalls

1. **Root-causing before mitigating** while users are still affected.
2. **Changing several things at once** — you won't know what fixed it.
3. **Trusting timestamp order across hosts**.
4. **Debugging with averages** — the problem lives in a slice (one region, one version, one tenant).
5. **No preserved evidence** — restarting everything destroys the state you needed (take dumps/snapshots first if safe).
6. **Blame-oriented post-mortems** — people hide information; fix systems, not people.

## Quick Reference

```
symptom → scope (split by attributes) → what changed? → hypothesis → test with data → mitigate → root cause → prevent
metrics: is it broken? · traces: where? · logs (by trace id): what exactly? · profiles/dumps: why is it stuck/hot?
check early: recent changes · timeouts/retries · exhaustion · data-dependent · clocks · caches · version skew · DNS/certs
```