Replication

1The Basic Shape: Leader and Followers

Database replication keeps a copy of the same data on more than one server. The dominant topology is leader-follower (also called primary-replica, or the older terms master-slave): one node, the leader, accepts all writes; one or more followers (replicas) receive a continuous stream of the leader's changes and apply them locally, ending up with (eventually, or immediately — see below) the same data.

        writes
Client ────────> [ LEADER ]
                      │  replication stream (WAL / binlog / oplog)
          ┌───────────┼───────────┐
          ▼           ▼           ▼
     [FOLLOWER 1] [FOLLOWER 2] [FOLLOWER 3]
          ▲
          │ reads can be served here too
       Client (read)

Followers replicate by consuming the leader's write-ahead log (PostgreSQL's WAL, MySQL's binlog, MongoDB's oplog) — the same durable log the leader already writes for crash recovery — rather than being sent already-executed higher-level SQL. Streaming the physical/logical log means a follower ends up in exactly the state the leader was in at each logged point, applied in the same order.

2Why Replicate: Two Distinct Motivations

Durability / availability — if the leader's disk dies, a follower has (up to replication lag) the same data and can be promoted to take over, instead of data being gone or the service being down until a backup is restored.

Read scaling — a single leader has a ceiling on how many reads its CPU/disk/memory can serve. Followers can serve read queries independently, so read throughput scales roughly linearly by adding more followers, without touching write capacity at all (writes must still all go through the one leader — replication does not help write scaling; that's what 03-sharding/ addresses).

These two motivations sometimes pull toward different configurations — durability wants confirmed replication before the leader reports success; read-scaling wants the leader unblocked as fast as possible — which is exactly the synchronous/asynchronous tradeoff below.

3Synchronous vs. Asynchronous Replication

Asynchronous:
  Client -> WRITE -> [LEADER] -- ack immediately --> Client
                          │
                          └── stream to follower(s), whenever it gets there

Synchronous:
  Client -> WRITE -> [LEADER] -- waits for follower ack --> ack to Client
                          │                    ▲
                          └── stream to follower ┘

Asynchronous replication: the leader commits and acknowledges the write to the client immediately, without waiting for any follower to confirm receipt. Lowest write latency. The cost: if the leader crashes before a follower has caught up, the most recent writes that were acknowledged to the client can be lost — they exist only on the now-dead leader.

Synchronous replication: the leader waits for at least one follower (or a specified quorum of followers) to confirm the write before acknowledging it to the client. No data loss on leader failure (the confirmed follower has everything that was acknowledged), at the cost of every write now waiting on a network round trip to a follower — higher and less predictable write latency, and if that follower is unreachable, writes can stall entirely unless the system is designed to fail over the synchronous role.

Most production systems use a middle ground: asynchronous replication to most followers (fast, accepted risk of a small loss window on leader crash), with one follower configured as synchronous specifically to bound the worst case (PostgreSQL's synchronous_standby_names, for example) — trading a small amount of latency for a hard guarantee that at least one up-to-date copy exists at all times.

4Read Replicas and Replication Lag

Directing read traffic to followers is called using read replicas. The catch is replication lag: the gap in time between a write committing on the leader and that write becoming visible on a given follower. Under async replication this gap is normally milliseconds, but under load, network issues, or a slow follower, it can grow to seconds or more.

t=0    write X=5 commits on LEADER
t=0    client is told "success"
t=0    client immediately reads X from a FOLLOWER
t=0    follower hasn't received the replication stream entry yet -> reads X=4 (STALE)
t=8ms  replication catches up -> follower now reads X=5

This is the classic read-your-own-writes problem: a user who just updated their profile, then immediately reloads the page and gets routed to a lagging replica, sees their own change appear to have not saved. Common mitigations: route a user's own post-write reads to the leader for some short window; track a log position/timestamp per write and only read from a follower that has caught up past it; or accept the staleness for read paths where it's tolerable (a public feed, an analytics dashboard) while routing read-after-write-sensitive paths to the leader unconditionally.

5Failover

When the leader becomes unavailable, one follower must be promoted to the new leader so writes can resume. The mechanics:

1. Detect the leader is actually down (not just slow or network-partitioned from one monitor — a false-positive failover that promotes a follower while the old leader is still alive and accepting writes creates two leaders accepting conflicting writes, a split-brain scenario).
2. Select which follower to promote — normally the one with the most up-to-date replication position, to minimize data loss.
3. Promote that follower (it stops following and starts accepting writes).
4. Repoint clients and the remaining followers at the new leader.

Manual failover is slow (minutes, involving a human) but safe against false positives. Automated failover (tools like Patroni for PostgreSQL, or managed-cloud-database built-in failover) is fast (seconds) but requires careful split-brain prevention — typically a consensus mechanism (a majority quorum, or an external coordinator like etcd/ZooKeeper) so promotion only happens when there's agreement that the old leader is genuinely gone, not just unreachable from one vantage point.

⚠️ Common Pitfalls

1. Reading from a replica immediately after writing and expecting to see the write — the classic read-your-own-writes staleness bug; route write-then-immediately-read paths to the leader, or track replication position explicitly.
2. Assuming asynchronous replication never loses data — it can, specifically the writes acknowledged to a client but not yet streamed to any surviving follower at the moment the leader dies.
3. Treating replication as a backup strategy — a replica faithfully replicates a bad write (a bug that corrupts data, or an accidental DELETE) just as fast as a good one; replicas protect against hardware/node failure, not against application-level mistakes, which is what point-in-time backups are for.
4. Automated failover without split-brain protection — promoting a follower during a transient network partition while the old leader is actually still up and serving writes, producing two simultaneously-writable leaders whose data will conflict.
5. Scaling reads onto replicas without measuring lag — adding replicas increases read capacity but does nothing for write capacity, and pushing more read traffic onto an already-lagging replica just adds load without shrinking the (write-bound) lag.

📋 Quick Reference

Leader-follower: one leader accepts writes, N followers replicate its write-ahead log
Motivations: durability/availability (survive leader loss) + read scaling (not write scaling)

Async replication: leader acks immediately, fast, risk of data loss on leader crash
Sync replication:  leader waits for follower ack, no loss, higher/less predictable latency
Common middle ground: async to most followers, sync to one, for a bounded worst case

Replication lag: time gap between a write committing on leader and appearing on a follower
Read-your-own-writes bug: reading a follower right after writing to the leader -> stale read

Failover: detect leader down -> pick most-caught-up follower -> promote -> repoint clients
Split-brain: two nodes both think they're leader -> conflicting writes -> needs quorum/consensus

Replication is NOT a backup: bad writes replicate too; use point-in-time backups separately

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dbscale_replication_theory.md · generated by tools/build_lessons.py