Distributed Transactions

This topic assumes familiarity with ACID transactions in a single database (databases/fundamentals/03-transactions_and_acid). Part 1 covers why that single-database guarantee breaks down the moment a business operation spans multiple services/databases, and the classic (largely legacy) answer, two-phase commit. Part 2 covers the pattern that replaced it in most modern distributed systems: the Saga.

1The Problem: One Business Operation, Multiple Databases

A single ACID transaction gives you atomicity — every statement inside BEGIN/COMMIT either all takes effect or none does — but that guarantee is scoped to one database connection talking to one database. The moment a business operation needs to update state in two separate databases (or two separate services, each owning its own database — the normal shape of a microservice architecture), there is no single BEGIN/COMMIT that can wrap both:

Order Service:  BEGIN; INSERT INTO orders (...); COMMIT;      -- its own DB
Payment Service: BEGIN; UPDATE accounts SET balance = ...; COMMIT;  -- a DIFFERENT DB

Placing an order needs BOTH to succeed together. But they are two independent
transactions against two independent databases -- nothing forces them to
succeed or fail as one atomic unit. If the order insert commits and the
payment charge then fails, you have an order with no payment. If it's the
other way around, you've charged a customer for an order that doesn't exist.

This is the core problem distributed transactions address: how do you get an atomicity-like guarantee ("all of these operations across multiple services happen, or none of them do") when there's no single database transaction that can enforce it.

2Two-Phase Commit (2PC): The Classic Answer

Two-phase commit introduces a coordinator that orchestrates all the participating databases/services (participants) through two distinct phases, so that either every participant commits or every participant aborts — never a mix.

Phase 1 (PREPARE / VOTE):
  Coordinator -> every participant: "can you commit this?"
  Each participant does all the work needed to GUARANTEE it CAN commit if
  told to (locks acquired, changes written to a durable log, but NOT yet
  visible/committed) and replies "yes" (vote-commit) or "no" (vote-abort).

Phase 2 (COMMIT / ABORT):
  IF every participant voted yes:
    Coordinator -> every participant: "COMMIT" -> each makes its prepared
    changes permanent and visible.
  IF any participant voted no (or didn't respond):
    Coordinator -> every participant: "ABORT" -> each rolls back.

   COORDINATOR                    PARTICIPANT A          PARTICIPANT B
        │── PREPARE ────────────────▶│                       │
        │── PREPARE ─────────────────┼───────────────────────▶│
        │◀── vote: yes ───────────────│                       │
        │◀── vote: yes ────────────────────────────────────────│
        │  (all voted yes -- safe to commit)
        │── COMMIT ──────────────────▶│                       │
        │── COMMIT ───────────────────┼───────────────────────▶│

The critical property Phase 1 buys: once a participant votes "yes," it has made an unconditional promise — it must be able to commit later no matter what, even across its own crash and restart, because the coordinator's final decision might come after a delay. This is why Phase 1 requires participants to durably persist their prepared state before voting yes, not just check that committing would be possible.

32PC's Fatal Weakness: The Blocking Coordinator

Two-phase commit's atomicity guarantee has one very costly dependency: the coordinator must stay reachable and eventually decide. Consider what happens if the coordinator crashes after Phase 1 but before sending its Phase 2 decision:

Coordinator: sent PREPARE to A and B, both voted yes...
Coordinator: CRASHES before sending COMMIT or ABORT to anyone

Participant A: has voted yes, has WRITTEN AND LOCKED its prepared changes,
               and now has NO IDEA whether the final decision was commit
               or abort -- it cannot safely commit (the coordinator might
               have decided abort after hearing from B) and cannot safely
               abort either (the coordinator might have decided commit).
               A is stuck HOLDING ITS LOCKS, indefinitely, until the
               coordinator recovers and tells it what happened.

This is 2PC's defining weakness: a participant that has voted yes but hasn't yet heard the final decision is blocked — it must hold its locks (meaning any other transaction touching that same data is also blocked) until the coordinator recovers, which could be a long time, or forever if the coordinator's crash is unrecoverable. A protocol meant to coordinate distributed work ends up making every participant's availability depend on one coordinator's availability — exactly the kind of single point of failure a distributed system is usually trying to avoid.

4Why 2PC Is Rarely Used Across Microservices Today

Beyond the blocking problem, 2PC has practical costs that make it a poor fit for the normal microservice shape (independently-owned databases, services that may be built with different technologies, higher latency between services than within one datacenter):

• It requires every participant to support the 2PC protocol (locking + durable prepared state) — most application databases used by microservices (and virtually all external third-party APIs) don't expose this.
• Holding locks across the full round-trip of coordinator <-> every participant <-> coordinator, over a network, for the duration of the entire operation, is far slower and more contention-prone than holding a lock for the duration of one local transaction.
• It fundamentally trades availability for consistency during a coordinator failure — which is often the opposite of what a microservice architecture is built to prioritize (see the upcoming CAP theorem topic).

2PC still shows up within a single, tightly-coupled system (some message-broker/database combinations use it internally, for instance), but it is rarely the answer for coordinating independently-owned services. Part 2 covers the pattern that took its place.

⚠️ Common Pitfalls

1. Assuming a "distributed transaction" gives the same guarantees as a local one — 2PC provides atomicity of outcome (all commit or all abort), but at the cost of availability during coordinator failure, which a single-database transaction never has to worry about.
2. Treating the prepare vote as a "check," not a promise — once a participant votes yes, it MUST be able to commit later, even after its own crash/restart; this is why prepared state has to be durably persisted, not just verified in memory.
3. Underestimating the blocking window — participants holding locks from PREPARE until they hear the final decision isn't a rare edge case, it's the expected steady-state cost of every 2PC transaction, and it directly limits throughput.
4. Reaching for 2PC as the default answer for "operation spans two services" — its blocking/availability cost and requirement that every participant implement the protocol make it a poor default fit for typical independently-owned microservice databases; Part 2's Saga pattern is the far more common real-world answer.

📋 Quick Reference

Problem: one business operation spans multiple independently-committed databases/services
         -- no single BEGIN/COMMIT can wrap all of them atomically

2PC Phase 1 (PREPARE): coordinator asks every participant "can you commit?" -- each
                        participant durably persists its prepared state and votes yes/no
2PC Phase 2 (COMMIT/ABORT): if ALL voted yes -> commit everywhere; if ANY voted no -> abort everywhere

Fatal weakness: a participant that voted yes MUST hold its locks until it hears the final
                decision -- if the coordinator crashes in between, that participant is
                BLOCKED, indefinitely, unable to safely commit or abort on its own

Part 1 covered why 2PC's blocking coordinator makes it a poor fit for typical microservice architectures. This part covers the pattern that replaced it in practice: the Saga, which trades strict atomicity for availability and eventual consistency.

1The Core Idea: A Sequence of Local Transactions, Not One Distributed One

A Saga breaks one business operation into a sequence of independent local transactions, each fully committed on its own, one per service — with no cross-service locks held at any point, and no coordinator blocking anyone:

Order Saga:
  Step 1: Order Service   -> BEGIN; create order (status=PENDING); COMMIT.  (local, done)
  Step 2: Payment Service -> BEGIN; charge card; COMMIT.                    (local, done)
  Step 3: Inventory Service -> BEGIN; reserve stock; COMMIT.                (local, done)
  Step 4: Order Service   -> BEGIN; update order (status=CONFIRMED); COMMIT. (local, done)

Each step is a completely ordinary, fully-committed local transaction — the moment Step 1 commits, the order genuinely exists in the Order Service's database, visible to anything else querying it, with no lock held waiting for Step 2 or Step 3 to happen. This is the fundamental departure from 2PC: nothing is held in a "prepared but uncommitted" limbo state waiting on a coordinator's final word.

2The Cost of That Freedom: Compensating Transactions

Committing each step independently means there's no way to "roll back" a later failure the way a single database transaction would — Step 1's commit is final and durable the instant it happens. If Step 3 (reserve stock) fails after Steps 1 and 2 have already committed, the Saga can't undo Step 2's card charge with a database ROLLBACK — that transaction is long since finished. Instead, the Saga must run an explicit compensating transaction: a new, separate operation whose entire purpose is to semantically undo the effect of an earlier step.

Order Saga, Step 3 (reserve stock) FAILS:
  Compensate Step 2: Payment Service -> BEGIN; refund the charge; COMMIT.
  Compensate Step 1: Order Service   -> BEGIN; update order (status=CANCELLED); COMMIT.
                                          (NOT delete the row -- see Section 4)

A compensating transaction is itself just an ordinary local transaction — but it's forward-moving business logic ("issue a refund," "cancel the order"), not a database-level undo. This means every step in a Saga that has a side effect worth compensating needs a corresponding compensating action designed in advance, as part of building the Saga, not something the database gives you for free.

3Two Ways to Coordinate a Saga: Choreography vs Orchestration

Choreography — no central coordinator; each service reacts to events published by the previous step and publishes its own event when done:

Order Service: creates order -> publishes OrderCreated
Payment Service: listens for OrderCreated -> charges card -> publishes PaymentCharged
Inventory Service: listens for PaymentCharged -> reserves stock -> publishes StockReserved
Order Service: listens for StockReserved -> confirms order

If Inventory Service instead publishes StockReservationFailed:
  Payment Service listens for THAT -> runs its refund compensation -> publishes PaymentRefunded
  Order Service listens for THAT -> cancels the order

Choreography keeps each service simple and fully decoupled (no service needs to know the whole saga's shape), but the overall flow of "what happens in what order" is implicit, scattered across every service's event handlers — understanding or debugging the full saga means tracing events through every participant's code.

Orchestration — a central orchestrator explicitly calls each step and explicitly triggers compensation on failure:

async function runOrderSaga(orderId: string) {
  await orderService.create(orderId);
  try {
    await paymentService.charge(orderId);
    try {
      await inventoryService.reserve(orderId);
    } catch {
      await paymentService.refund(orderId);       // compensate payment
      throw new Error("saga failed at inventory step");
    }
  } catch {
    await orderService.cancel(orderId);            // compensate order creation
    throw new Error("saga failed");
  }
  await orderService.confirm(orderId);
}

Orchestration makes the full sequence (and its compensation logic) explicit and readable in one place, at the cost of introducing a central component that every step depends on — not a blocking dependency like 2PC's coordinator (each step still commits independently and doesn't hold locks waiting on the orchestrator), but a component whose own reliability and correctness the whole saga now depends on.

4Why Sagas Only Give You Eventual Consistency

Between Step 1 committing and Step 4 committing, the system is observably inconsistent from a strict standpoint — an order can exist with status PENDING while payment hasn't been charged yet, visible to anything querying the Order Service in that window. This is the fundamental tradeoff a Saga makes versus 2PC's all-or-nothing atomicity: a Saga guarantees the system will eventually reach a consistent end state (either fully confirmed, or fully compensated back to cancelled) — but not that every intermediate moment looks consistent to an outside observer. This is precisely why Step 1's compensation in Section 2 updates the order to CANCELLED rather than deleting it: other parts of the system may have already read and acted on the PENDING order in that intermediate window, and a visible, explicit CANCELLED state lets them react correctly, whereas a silent delete would leave them with no explanation for a customer-facing order that simply vanished.

⚠️ Common Pitfalls

1. Forgetting that compensation is forward-moving logic, not a rollback — a compensating transaction (refund, cancel, release) has to be designed and built for every compensable step; the database gives you none of this for free the way ROLLBACK does inside one transaction.
2. Deleting/hard-reverting state during compensation instead of marking it explicitly failed/cancelled — other parts of the system may have already observed and acted on the intermediate state; an explicit terminal status lets them react correctly instead of encountering data that silently vanished.
3. Choreography sprawl — as a saga grows more steps, purely event-driven choreography with no central view becomes very hard to trace end-to-end; teams often migrate a complex saga to orchestration specifically for that visibility.
4. Assuming compensations always succeed — a refund call can itself fail; production saga implementations need their OWN retry/backoff (previous topic) around compensating steps, and a plan for what happens if compensation itself can't complete.
5. Expecting saga-based consistency to look like ACID from the outside — code that reads state mid-saga (e.g., a PENDING order) needs to be written with that intermediate visibility in mind; treating a Saga's eventual consistency as if it were the same guarantee as a single ACID transaction leads to bugs where "impossible" intermediate states are observed and mishandled.

📋 Quick Reference

Saga = a sequence of independently-COMMITTED local transactions, one per service --
       no cross-service locks held, no blocking coordinator (contrast with 2PC)

On failure: run COMPENSATING transactions for already-committed steps, in reverse order
            (compensation = new forward-moving undo logic: refund, cancel, release --
             NOT a database rollback, since the original steps are already committed)

Choreography: services react to each other's events, no central coordinator -- simple per
              service, but the overall flow is implicit and scattered
Orchestration: a central orchestrator calls each step and its compensation explicitly --
               visible end-to-end flow, but a component the whole saga depends on

Guarantee: EVENTUAL consistency (will reach a consistent end state), NOT atomicity
           (intermediate states ARE observable to the rest of the system)

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dist_distributed_transactions_theory_1.md, dist_distributed_transactions_theory_2.md · generated by tools/build_lessons.py