Service-to-Service Communication

1The Two Fundamental Styles

Once functionality lives in separate processes (see Monolith vs Microservices), every interaction between them has to pick one of two fundamentally different communication styles:

• Synchronous — the caller sends a request and blocks (waits) until a response arrives before continuing.
• Asynchronous — the caller sends something and continues immediately, without waiting for the receiver to finish (or even start) processing it.

This single choice — sync or async, for each interaction — is the most consequential design decision in a distributed system, because it determines coupling, latency behavior, and failure modes for that call. Nothing else in this topic matters as much as getting this choice right per interaction; see Under the Hood for exactly why.

2Synchronous Communication: REST

REST over HTTP is the default synchronous style: a client sends an HTTP request to a resource-oriented URL, and the server replies with a status code and body.

// orders-service calling inventory-service synchronously over REST
async function reserveStock(sku: string, qty: number): Promise<boolean> {
  const res = await fetch(`http://inventory-service/reservations`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ sku, qty }),
    signal: AbortSignal.timeout(2000), // never call synchronously without a timeout
  });
  return res.ok;
}

REST's strengths: human-readable, cacheable (via HTTP semantics), universally supported, easy to debug with a browser or curl. Its weakness for service-to-service traffic: JSON serialization overhead, no compile-time contract (a typo in a field name fails at runtime, not build time), and no native support for streaming.

3Synchronous Communication: gRPC

gRPC is a binary RPC framework built on HTTP/2 and Protocol Buffers. Services define a contract in a .proto file, and gRPC generates strongly-typed client/server code in every supported language from that single definition.

// inventory.proto
service InventoryService {
  rpc ReserveStock (ReserveRequest) returns (ReserveResponse);
}
message ReserveRequest { string sku = 1; int32 qty = 2; }
message ReserveResponse { bool reserved = 1; }

// generated client usage looks like a normal async function call
const response = await inventoryClient.reserveStock({ sku: "ABC", qty: 3 });

gRPC advantages over REST/JSON: smaller payloads (binary Protobuf vs text JSON), a compiler-checked contract shared across languages, and native support for streaming RPCs (client-streaming, server-streaming, bidirectional). Cost: not human-readable on the wire, harder to debug ad hoc, less friendly to public/browser-facing APIs.

4Asynchronous Communication: Messaging

Instead of calling another service directly, a service publishes a message to a broker (see the Message Brokers group) and moves on immediately.

// orders-service publishing an event asynchronously — does not wait for
// inventory-service, email-service, or analytics-service to do anything
async function placeOrder(order: Order) {
  await db.insertOrder(order);
  await broker.publish("orders.created", { orderId: order.id, items: order.items });
  return { status: "accepted" }; // caller gets a response before downstream work happens
}

Every interested service (inventory, email, analytics) consumes the event independently, at its own pace, with no code change required in orders-service to add a new subscriber. This is the async style's core property: zero temporal coupling — publisher and consumer(s) don't need to be online at the same moment.

5Comparing the Styles Directly

6Request/Response Over Async Transport

Sync and async aren't always REST-vs-messaging — you can build a request/response pattern on top of an async transport, trading immediate blocking for a correlation ID:

// caller publishes a request with a replyTo queue and a correlationId,
// then waits (with a timeout) on that specific reply queue
async function askInventoryAsync(sku: string): Promise<number> {
  const correlationId = crypto.randomUUID();
  await broker.publish("inventory.query", { sku, replyTo: "orders.replies", correlationId });
  return await broker.awaitReply("orders.replies", correlationId, { timeoutMs: 3000 });
}

This still blocks the caller (so it behaves synchronously from the caller's point of view) but decouples the transport — useful when you want messaging's durability/retry semantics but still need an answer before continuing.

7Choosing a Style Per Interaction

The decision is made per interaction, not once per system — a real service typically uses all three:

• gRPC for internal, high-throughput, latency-sensitive service-to-service calls where both sides are internal and you control both ends.
• REST for public-facing APIs, or internal calls where human-debuggability and broad tooling support matter more than raw efficiency.
• Messaging for anything that can happen later, needs to fan out to multiple consumers, or must survive the callee being temporarily down.

A single user action (placing an order) commonly mixes all three: a synchronous REST call from the browser to orders-service, a synchronous gRPC call from orders-service to inventory-service to check stock right now, and an asynchronous event published for email-service and analytics-service to consume whenever they're ready.

orders-service --HTTP request--> inventory-service
     |                                  |
     | blocked, holding a thread/       | must be reachable, healthy,
     | connection, accumulating         | and respond within the
     | latency the whole time           | caller's timeout budget
     v                                  v
  caller's total latency = its own work + full round trip to callee
  caller's success rate  <= callee's success rate (can never exceed it)

orders-service --publish--> [ broker, durable ] --(later, independently)--> inventory-service
     |
     v
  caller's latency = just the publish call (broker ack), NOT inventory-service's processing time
  caller's success  = "message durably accepted by broker", NOT "inventory-service finished"

⚠️ Common Pitfalls

1. Calling a chain of five synchronous services for one user request — latency adds up hop by hop, and the whole chain's availability is bounded by the product of every hop's uptime; a single slow service in the middle stalls everything behind it.
2. Using messaging for something that needs an immediate answer — a "check card validity" call published as a fire-and-forget event with no reply path leaves the caller with nothing to show the user.
3. No timeout on a synchronous call — without one, a hung callee hangs the caller forever (or until an OS-level socket timeout, which is usually far too long); see the Retry and Circuit Breaker topics for what belongs alongside a timeout.
4. Treating "message published" as "work done" — a caller that publishes and immediately tells the user "success" is lying if the downstream consumer can still fail; either design the UX around eventual consistency or add a mechanism to report back.
5. Picking gRPC or REST as a blanket policy for the whole system — the right choice is per interaction; a system that forces every internal call to be synchronous REST when half of them could be fire-and-forget events pays unnecessary coupling costs everywhere.

📋 Quick Reference

Synchronous (REST/gRPC): caller blocks, needs answer NOW, coupled availability + additive latency
Asynchronous (messaging): caller returns immediately, decoupled availability, outcome unknown at return time

REST:  human-readable, cacheable, universal tooling      -> public APIs, ad hoc debugging
gRPC:  binary, compiler-checked contract, streaming       -> internal, high-throughput, low-latency calls
Async: fan-out, durability, temporal decoupling            -> "can happen later," multiple subscribers

Chained sync calls: latency ADDS, availability MULTIPLIES (compounds downward)
Rule of thumb: sync when you need the answer now, async when the work can happen later

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dist_service_to_service_communication_theory.md · generated by tools/build_lessons.py