Scalability Patterns

1Vertical vs Horizontal Scaling

Vertical scaling (scaling up) means giving one instance more resources — more CPU, more RAM, a faster disk. Horizontal scaling (scaling out) means adding more instances and spreading load across them.

Vertical:    [1 server: 4 CPU, 16GB]  ->  [1 server: 32 CPU, 128GB]
Horizontal:  [1 server]  ->  [1 server] [server] [server] [server]  (load balanced)

Vertical scaling is simpler (no distribution logic needed, nothing about the application has to change) but has a hard ceiling — there's a largest machine you can buy, at rapidly increasing cost per unit of additional capacity — and it does nothing for fault tolerance: one bigger server is still one server, still a single point of failure (see the previous topic). Horizontal scaling has no comparable hard ceiling (add another instance) and directly buys redundancy alongside capacity, but it only works cleanly if the thing being scaled can actually be split across multiple instances — which is precisely where the next section matters.

2Statelessness: The Precondition for Horizontal Scaling

A stateless service instance holds no data in memory that would be lost, or would differ from another instance, between requests — every request carries (or looks up externally) everything needed to handle it. This is what makes horizontal scaling trivial for the application tier: any instance can handle any request, so a load balancer can distribute traffic however it wants, and adding/removing instances is just a capacity change, not a data-migration problem.

// STATEFUL (bad for horizontal scaling): session stored in this process's memory
const sessions = new Map<string, UserSession>(); // lives ONLY on this one instance
app.get("/profile", (req, res) => {
  const session = sessions.get(req.cookies.sessionId); // a DIFFERENT instance has no idea this session exists
  res.json(session.user);
});

// STATELESS: session stored in a shared store every instance can reach
app.get("/profile", async (req, res) => {
  const session = await redisClient.get(req.cookies.sessionId); // any instance gets the same answer
  res.json(JSON.parse(session).user);
});

The stateful version breaks the moment a load balancer routes two requests from the same user to two different instances — one has their session, the other doesn't. The stateless version works identically no matter which instance handles the request, which is exactly the property that makes "just add more instances" a safe, mechanical scaling move rather than a data-consistency problem. This is why real systems push state out of the application tier entirely — into a database, a cache like Redis, or a dedicated session store — keeping the application instances themselves freely, cheaply horizontally scalable.

3Scaling the Data Layer: Read Replicas

The application tier being stateless doesn't automatically solve scaling for the database itself, which by definition DOES hold state. A common first step: read replicas — copies of the primary database that stay continuously updated (via replication) and serve read-only queries, while writes still go to the single primary:

                    writes
Application  ────────────────────▶  [Primary DB]
                                          │ replication (continuous)
                    reads                ▼
Application  ─────────────────▶  [Read Replica 1] [Read Replica 2] [Read Replica 3]

This works well when a system is read-heavy (true of most web applications — far more page views than writes), letting read capacity scale horizontally (add more replicas) while writes remain centralized and simple. The tradeoff: replicas usually lag the primary by some small amount (replication isn't instantaneous), so a read immediately after a write can return stale data unless the application specifically routes that read to the primary — an eventual-consistency tradeoff, conceptually the same shape as the CAP theorem topic's AP choice, applied specifically to read-replica lag.

4Scaling the Data Layer: Sharding/Partitioning

Read replicas don't help with write throughput — every replica still receives the same writes from the one primary, so the primary's write capacity is still a ceiling. Sharding (partitioning) splits the data itself across multiple independent database instances, each owning a subset of the data, so writes for different shards can happen in parallel on entirely separate machines:

Shard by user_id range:
  [Shard A: users 1-1,000,000]     [Shard B: users 1,000,001-2,000,000]     [Shard C: ...]

A write for user 500,000 only touches Shard A's database -- it never contends
with a write for user 1,500,000 on Shard B. Total write throughput scales
with the NUMBER of shards, not the capacity of one machine.

Sharding buys real horizontal write scalability, at real cost: queries that need to span multiple shards (e.g. "find all users across the whole system matching X") become significantly harder, and choosing a shard key that distributes load evenly (avoiding a "hot shard" that gets disproportionate traffic) is a real design problem, not an afterthought.

5Caching Layers

Adding a cache in front of a database (or an expensive computation) absorbs read traffic before it ever reaches the underlying data store, which is often the cheapest, highest-leverage scaling move available for read-heavy workloads — a cache hit costs a fraction of a full database query, and every cache hit is one query the database never has to handle at all. The full mechanics (invalidation strategies, TTLs, cache-aside vs write-through) belong to the dedicated caching topics in this curriculum (databases/caching/) — the scaling-relevant point here is narrower: caching doesn't just make individual requests faster, it directly reduces the load the data layer itself needs to scale to handle, which is often a cheaper lever to pull than scaling the database horizontally at all.

6Queue-Based Load Leveling

For work that doesn't need to happen synchronously within a request/response cycle, putting a queue between the producer of work and the thing that processes it decouples the rate work arrives from the rate it's processed, letting the processing side scale independently and absorb bursts without the producer-facing request ever slowing down:

Request arrives -> API enqueues a job (fast, returns immediately) -> responds to caller
                                    │
                                    ▼
                            [Job Queue]  <- absorbs bursts; this is exactly the
                                    │        earlier Backpressure topic's bounded
                                    ▼        queue, applied here as a scaling tool
                        [N worker instances, scaled independently of the API tier]

This is the same queue-based decoupling from the Backpressure topic, viewed from a scaling angle rather than a stability angle: because the API tier's job is now just "accept work and enqueue it" (fast, cheap, stateless), it can be scaled independently of the worker tier, and the worker tier's instance count can be scaled based on queue depth (more backlog -> spin up more workers) rather than needing to match the API tier's instance count at all.

7How These Patterns Compose

None of these patterns are mutually exclusive — a mature, scaled system typically layers several together: a stateless, horizontally-scaled API tier, sitting behind a cache that absorbs most read traffic, backed by a primary database with read replicas for the remaining read load, sharded if write volume demands it, with a queue decoupling any asynchronous/bursty work into an independently-scaled worker tier — with circuit breakers and bulkheads (previous topics) protecting each of those layers from cascading failures in the others, and backpressure (an earlier topic) governing what happens when even that scaled-out capacity is temporarily exceeded.

⚠️ Common Pitfalls

1. Relying on vertical scaling indefinitely — it has a real ceiling, rapidly worsening cost-per-unit-capacity as you approach it, and buys zero fault tolerance no matter how large the single machine gets.
2. Keeping state in application-instance memory "just for this one feature" — silently breaks horizontal scaling for exactly that feature the moment a load balancer routes two related requests to two different instances.
3. Adding read replicas to fix a WRITE throughput problem — replicas only scale reads; a write-bottlenecked system needs sharding (or a fundamentally different write path), not more replicas.
4. Sharding on a key that produces a hot shard — e.g. sharding by signup date when a huge fraction of active users signed up recently concentrates load on one shard, defeating the purpose of splitting the data at all.
5. Treating a queue as a magic scaling fix without addressing what happens when the queue itself backs up — a queue absorbs bursts, but an under-scaled worker tier behind it still needs the same backpressure discipline from the earlier topic; a queue with no bound and no worker autoscaling just relocates the overload problem, it doesn't solve it.

📋 Quick Reference

Vertical scaling: bigger machine -- simple, hard ceiling, zero fault-tolerance benefit
Horizontal scaling: more machines -- requires statelessness, no comparable ceiling, buys redundancy

Statelessness = precondition for horizontal app-tier scaling: no per-instance memory state,
                shared external store (DB/cache/session store) instead

Read replicas: scale READS, not writes; replicas lag the primary (eventual consistency)
Sharding: scale WRITES by splitting data across independent DBs; cross-shard queries get harder
Caching: absorbs read load before it reaches the data layer at all -- often the cheapest lever
Queue-based load leveling: decouples work arrival rate from processing rate, scales independently

Real systems LAYER these together, with backpressure/circuit-breaking/bulkheading protecting
the seams between layers

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dist_scalability_patterns_theory.md · generated by tools/build_lessons.py