Debugging Distributed Systems

1Why It's Different

In a single process you can set a breakpoint, step through, and see the whole state. In a distributed system:

• the "program" spans many processes, machines and teams;
• failures are partial (some instances, some regions, some requests);
• timing matters — bugs depend on interleavings, latency, retries, clock skew;
• you usually can't reproduce production conditions locally;
• the act of observing (adding logs, attaching debuggers) can change timing.

So debugging becomes investigation from telemetry, guided by hypotheses — closer to science than to stepping through code.

2A Method

1. define the symptom precisely   what fails? for whom? since when? how much? (error rate, latency, affected %)
2. scope it                        which endpoints/regions/versions/tenants/instances? which not?
3. look for what changed           deploys, config/flag changes, traffic shape, dependencies, infra events
4. form a hypothesis               "requests from app v4.2 to /checkout time out on payments since 14:05"
5. test it with data               dashboards, traces, logs filtered by the hypothesis; compare good vs bad
6. mitigate first                  rollback, disable flag, shift traffic, scale — restore service before root-causing
7. find the root cause             5 whys / causal analysis on the preserved evidence
8. prevent recurrence              fix + tests + alert/runbook improvements (blameless post-mortem)

Compare good with bad: the fastest way to localize a problem is to split traffic by attributes (version, region, instance, customer, endpoint, payload size) and find the dimension where errors concentrate. High-cardinality, structured telemetry (devops-cloud/observability/) is what makes that possible.

3The Tools and What They Answer

4Common Culprits (Check These Early)

1. A recent change — deploy, config, feature flag, dependency upgrade, infra change. (A large share of incidents.)
2. Timeouts and retries interacting (06-…, 07-…) — amplification, metastable failures.
3. Resource exhaustion — connection pools, file descriptors, memory, thread pools, disk, rate-limit quotas.
4. Data-dependent bugs — one tenant's huge payload, one malformed record (poison message), unicode/timezones.
5. Clock and ordering — skew between hosts, events out of order, TTLs and token expiry.
6. Caching — stale or poisoned entries, cache stampedes.
7. Partial deploys / version skew — old and new versions talking with incompatible schemas.
8. DNS and certificates — expired certs, stale DNS caches.

5Distributed-System Bug Patterns

• Heisenbugs — disappear when observed (timing). Use always-on structured telemetry rather than ad-hoc debug logging.
• Split brain — two nodes both think they're leader (check fencing tokens, quorum).
• Lost updates — concurrent writers (02-race_conditions/), check versions/CAS.
• Duplicate processing — at-least-once delivery without idempotency.
• Thundering herds — synchronized retries, cache expiry, cron jobs at :00.
• Cascading failures — one slow dependency saturating callers (07-resilience_and_reliability_patterns/).

6Making Systems Debuggable (Before You Need It)

• Structured logs with trace IDs, service version, region, tenant (devops-cloud/observability/01-logging).
• RED metrics per endpoint and per dependency; saturation metrics for every pool and queue.
• Tracing with sampling that keeps errors and slow requests.
• Deploy/flag events as annotations on dashboards.
• Runbooks per alert; dashboards that start from the user symptom.
• Debug endpoints (protected): build info, config (redacted), pool stats, feature flags.
• The ability to turn up logging for one tenant/request at runtime without a deploy.

Under the HoodWhy Clocks Make Causality Hard▾

Timestamps from different machines aren't directly comparable: NTP keeps clocks within milliseconds normally, but skew of tens or hundreds of ms happens, and a VM pause or leap-second smear can move clocks unexpectedly. Sorting a merged log by wall-clock time can therefore show an effect before its cause. Distributed systems establish causal order differently: Lamport timestamps (a counter incremented on every event and set to max(local, received) + 1 on message receipt) give an order consistent with causality; vector clocks detect concurrency; and tracing's parent/child span links encode causality explicitly, which is why traces are more trustworthy than timestamp-sorted logs when reconstructing "what caused what". When comparing timings across hosts, prefer durations measured on one host (span durations) to differences between two hosts' timestamps.

⚠️ Common Pitfalls

1. Root-causing before mitigating while users are still affected.
2. Changing several things at once — you won't know what fixed it.
3. Trusting timestamp order across hosts.
4. Debugging with averages — the problem lives in a slice (one region, one version, one tenant).
5. No preserved evidence — restarting everything destroys the state you needed (take dumps/snapshots first if safe).
6. Blame-oriented post-mortems — people hide information; fix systems, not people.

📋 Quick Reference

symptom → scope (split by attributes) → what changed? → hypothesis → test with data → mitigate → root cause → prevent
metrics: is it broken? · traces: where? · logs (by trace id): what exactly? · profiles/dumps: why is it stuck/hot?
check early: recent changes · timeouts/retries · exhaustion · data-dependent · clocks · caches · version skew · DNS/certs

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: adv_debugging_distributed_systems_theory.md · generated by tools/build_lessons.py