# Service-to-Service Communication

## 1. The Two Fundamental Styles

Once functionality lives in separate processes (see *Monolith vs Microservices*), every interaction between them has to pick one of two fundamentally different communication styles:

- **Synchronous** — the caller sends a request and blocks (waits) until a response arrives before continuing.
- **Asynchronous** — the caller sends something and continues immediately, without waiting for the receiver to finish (or even start) processing it.

This single choice — sync or async, for each interaction — is the most consequential design decision in a distributed system, because it determines coupling, latency behavior, and failure modes for that call. Nothing else in this topic matters as much as getting this choice right per interaction; see Under the Hood for exactly why.

## 2. Synchronous Communication: REST

**REST over HTTP** is the default synchronous style: a client sends an HTTP request to a resource-oriented URL, and the server replies with a status code and body.

```ts
// orders-service calling inventory-service synchronously over REST
async function reserveStock(sku: string, qty: number): Promise<boolean> {
  const res = await fetch(`http://inventory-service/reservations`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ sku, qty }),
    signal: AbortSignal.timeout(2000), // never call synchronously without a timeout
  });
  return res.ok;
}
```

REST's strengths: human-readable, cacheable (via HTTP semantics), universally supported, easy to debug with a browser or `curl`. Its weakness for service-to-service traffic: JSON serialization overhead, no compile-time contract (a typo in a field name fails at runtime, not build time), and no native support for streaming.

## 3. Synchronous Communication: gRPC

**gRPC** is a binary RPC framework built on HTTP/2 and Protocol Buffers. Services define a contract in a `.proto` file, and gRPC generates strongly-typed client/server code in every supported language from that single definition.

```protobuf
// inventory.proto
service InventoryService {
  rpc ReserveStock (ReserveRequest) returns (ReserveResponse);
}
message ReserveRequest { string sku = 1; int32 qty = 2; }
message ReserveResponse { bool reserved = 1; }
```

```ts
// generated client usage looks like a normal async function call
const response = await inventoryClient.reserveStock({ sku: "ABC", qty: 3 });
```

gRPC advantages over REST/JSON: smaller payloads (binary Protobuf vs text JSON), a compiler-checked contract shared across languages, and native support for streaming RPCs (client-streaming, server-streaming, bidirectional). Cost: not human-readable on the wire, harder to debug ad hoc, less friendly to public/browser-facing APIs.

## 4. Asynchronous Communication: Messaging

Instead of calling another service directly, a service publishes a message to a **broker** (see the Message Brokers group) and moves on immediately.

```ts
// orders-service publishing an event asynchronously — does not wait for
// inventory-service, email-service, or analytics-service to do anything
async function placeOrder(order: Order) {
  await db.insertOrder(order);
  await broker.publish("orders.created", { orderId: order.id, items: order.items });
  return { status: "accepted" }; // caller gets a response before downstream work happens
}
```

Every interested service (inventory, email, analytics) consumes the event independently, at its own pace, with no code change required in `orders-service` to add a new subscriber. This is the async style's core property: **zero temporal coupling** — publisher and consumer(s) don't need to be online at the same moment.

![Synchronous chain adds latency and multiplies-down availability hop by hop; asynchronous fan-out lets the caller return immediately while consumers process later at their own pace](diagrams/01-sync-vs-async.svg)

## 5. Comparing the Styles Directly

| Dimension | Synchronous (REST/gRPC) | Asynchronous (messaging) |
|---|---|---|
| Caller blocks? | Yes, until response or timeout | No, returns immediately after publish |
| Coupling | Caller must know callee's address, and callee must be up | Caller only needs to know the broker; callee can be down |
| Latency exposed to caller | Sum of every synchronous hop in the chain | Just the publish call — processing happens later |
| Best for | "I need the answer now" (read a price, validate a card) | "This can happen later" (send an email, update analytics) |
| Failure visibility | Immediate — caller gets an error or timeout right away | Delayed — a failure is invisible to the publisher; needs monitoring (see Dead Letter Queues) |
| Adding a new consumer | Requires changing the caller to add another call | Zero change to the publisher — just subscribe |
| Ordering/consistency | Caller sees a consistent result immediately | Eventually consistent — consumers catch up asynchronously |

## 6. Request/Response Over Async Transport

Sync and async aren't always REST-vs-messaging — you can build a request/response *pattern* on top of an async *transport*, trading immediate blocking for a correlation ID:

```ts
// caller publishes a request with a replyTo queue and a correlationId,
// then waits (with a timeout) on that specific reply queue
async function askInventoryAsync(sku: string): Promise<number> {
  const correlationId = crypto.randomUUID();
  await broker.publish("inventory.query", { sku, replyTo: "orders.replies", correlationId });
  return await broker.awaitReply("orders.replies", correlationId, { timeoutMs: 3000 });
}
```

This still blocks the caller (so it behaves synchronously from the caller's point of view) but decouples the transport — useful when you want messaging's durability/retry semantics but still need an answer before continuing.

## 7. Choosing a Style Per Interaction

The decision is made **per interaction**, not once per system — a real service typically uses all three:

- **gRPC** for internal, high-throughput, latency-sensitive service-to-service calls where both sides are internal and you control both ends.
- **REST** for public-facing APIs, or internal calls where human-debuggability and broad tooling support matter more than raw efficiency.
- **Messaging** for anything that can happen later, needs to fan out to multiple consumers, or must survive the callee being temporarily down.

A single user action (placing an order) commonly mixes all three: a synchronous REST call from the browser to `orders-service`, a synchronous gRPC call from `orders-service` to `inventory-service` to check stock *right now*, and an asynchronous event published for `email-service` and `analytics-service` to consume whenever they're ready.

## Under the Hood: The Coupling/Latency Tradeoff, Precisely

The reason "sync or async" is the single most consequential per-interaction decision is that it fixes two properties simultaneously, and you cannot get the good side of both for the same call.

### What synchronous coupling actually locks in

A synchronous call makes the caller's success **directly conditional** on the callee's availability, for the entire duration of the call:

```
orders-service --HTTP request--> inventory-service
     |                                  |
     | blocked, holding a thread/       | must be reachable, healthy,
     | connection, accumulating         | and respond within the
     | latency the whole time           | caller's timeout budget
     v                                  v
  caller's total latency = its own work + full round trip to callee
  caller's success rate  <= callee's success rate (can never exceed it)
```

This is not a design flaw to be engineered away — it is what "synchronous" *means*. If `inventory-service` is down, slow, or overloaded, that failure propagates directly and immediately into `orders-service`'s response time and error rate. Chain three synchronous calls (A calls B calls C calls D) and A's latency is the *sum* of every hop, and A's availability is bounded by the *product* of every service's uptime in the chain (99.9% × 99.9% × 99.9% ≈ 99.7%, a lower number than any individual service). This compounding is the concrete, measurable cost of synchronous coupling, and it's the reason long synchronous chains are treated as an architectural smell.

### What asynchronous decoupling actually buys — and what it costs

Publishing a message hands responsibility to the broker and returns immediately. The caller's latency and success rate are now decoupled from the consumer's:

```
orders-service --publish--> [ broker, durable ] --(later, independently)--> inventory-service
     |
     v
  caller's latency = just the publish call (broker ack), NOT inventory-service's processing time
  caller's success  = "message durably accepted by broker", NOT "inventory-service finished"
```

The caller no longer needs `inventory-service` to be up *at that instant* — the broker holds the message until a consumer is ready. But this buys decoupling at a real cost: the caller genuinely does not know, at the moment it returns "accepted" to its own caller, whether the downstream work has succeeded, is still pending, or will eventually fail and land in a dead-letter queue. Any workflow that needs to tell the *end user* "yes, this specific thing is done" cannot be fully async at that boundary — something in the system still has to become synchronous eventually (polling, a webhook, a WebSocket push) to close that loop.

### Why the choice can't be "always async" either

Async communication trades immediate consistency for throughput and resilience — but a caller that genuinely needs a fact *right now* (is this card valid, is this seat still available) cannot get that fact from a system designed to answer "eventually." Forcing a synchronous decision through an asynchronous pipe means either blocking on a reply anyway (Section 6's pattern, which reintroduces most of the sync latency/availability coupling it was meant to avoid) or answering with stale information. The tradeoff is fundamental, not a tooling limitation: **synchronous buys an immediate, consistent answer at the cost of coupling caller availability/latency to callee availability/latency; asynchronous buys decoupling at the cost of the caller not knowing the outcome at return time.** Every interaction in a distributed system is a deliberate pick of one side of that tradeoff.

## Common Pitfalls

1. **Calling a chain of five synchronous services for one user request** — latency adds up hop by hop, and the whole chain's availability is bounded by the product of every hop's uptime; a single slow service in the middle stalls everything behind it.
2. **Using messaging for something that needs an immediate answer** — a "check card validity" call published as a fire-and-forget event with no reply path leaves the caller with nothing to show the user.
3. **No timeout on a synchronous call** — without one, a hung callee hangs the caller forever (or until an OS-level socket timeout, which is usually far too long); see the Retry and Circuit Breaker topics for what belongs alongside a timeout.
4. **Treating "message published" as "work done"** — a caller that publishes and immediately tells the user "success" is lying if the downstream consumer can still fail; either design the UX around eventual consistency or add a mechanism to report back.
5. **Picking gRPC or REST as a blanket policy for the whole system** — the right choice is per interaction; a system that forces every internal call to be synchronous REST when half of them could be fire-and-forget events pays unnecessary coupling costs everywhere.

## Quick Reference

```
Synchronous (REST/gRPC): caller blocks, needs answer NOW, coupled availability + additive latency
Asynchronous (messaging): caller returns immediately, decoupled availability, outcome unknown at return time

REST:  human-readable, cacheable, universal tooling      -> public APIs, ad hoc debugging
gRPC:  binary, compiler-checked contract, streaming       -> internal, high-throughput, low-latency calls
Async: fan-out, durability, temporal decoupling            -> "can happen later," multiple subscribers

Chained sync calls: latency ADDS, availability MULTIPLIES (compounds downward)
Rule of thumb: sync when you need the answer now, async when the work can happen later
```