Scalability: Vertical vs Horizontal
Your service works for 100 users. What happens at 100,000? Learn the two ways to add capacity, scale up and scale out, see where each one hits a wall, and why stateless services are the key to growing horizontally.
🎯 What you'll learn
- The difference between vertical and horizontal scaling
- Why the cost of bigger machines curves upward
- Why horizontal scaling requires stateless services
- How real systems mix both approaches
📚 Before you start
1The Problem Scalability Solves
A system that works for 100 users can fall over at 100,000. Scalability is a system's ability to handle growing load — more users, more requests, more data — by adding resources, rather than by rewriting the system every time traffic doubles. There are exactly two axes along which you add resources: make the existing machine bigger (vertical scaling, "scale up"), or add more machines (horizontal scaling, "scale out"). Almost every other topic in this stack — load balancing, CDNs, database sharding, service discovery — exists to make horizontal scaling actually work.
PetPal's API runs on one 4 vCPU server handling 2,000 requests/second. A TV feature is scheduled for Friday and marketing expects 9,000 req/s.
- Option A (scale up): move to a 16 vCPU machine, with a reboot and maybe 7,500 req/s.
- Option B (scale out): run five 4 vCPU servers behind a load balancer: ~10,000 req/s, and losing one server only costs 20% of capacity.
The rest of this lesson explains when each option is right.
2Vertical Scaling — Scale Up
Vertical scaling means giving one machine more CPU, RAM, disk I/O, or network bandwidth.
t0: 1 server, 4 vCPU / 16 GB RAM → handles ~2,000 req/s
t1: 1 server, 16 vCPU / 64 GB RAM → handles ~7,500 req/s
t2: 1 server, 64 vCPU / 256 GB RAM → handles ~22,000 req/s (diminishing returns kick in)Pros:
- No code changes. A single-threaded bottleneck or an in-memory cache "just works" better on a bigger box.
- No distributed-systems complexity — no partitioning, no consistency questions, no need for a load balancer at all.
Cons:
- Hard ceiling. The largest cloud instances top out (e.g., AWS has instances with a few hundred vCPUs) — you cannot scale up forever.
- Single point of failure. One box means one box that can die and take everything down.
- Non-linear cost. Bigger instances cost more than proportionally more (see the cost curve in Under the Hood).
- Usually requires downtime or a failover to resize (you can't "add RAM" to a running VM in most clouds without a reboot).
3Horizontal Scaling — Scale Out
Horizontal scaling means adding more machines of (roughly) the same size and spreading load across them, usually behind a load balancer (see system-design/load_balancing_algorithms).
t0: 1 server, 4 vCPU each → 2,000 req/s total
t1: 4 servers, 4 vCPU each → ~8,000 req/s total (near-linear)
t2: 20 servers, 4 vCPU each → ~38,000 req/s total (slightly sub-linear — coordination overhead)Pros:
- No hard ceiling — in principle, add another node. Cloud auto-scaling groups exploit exactly this.
- Fault tolerance — one node dying takes out
1/Nof capacity, not all of it. Combine with health checks and the load balancer just routes around it. - Elasticity — scale out for a traffic spike (Black Friday), scale back in overnight, paying only for what you use.
Cons:
- Requires the application to be stateless (or to externalize state to a shared store like Redis or a database) — see
distributed-systems/fundamentals/scalability_patterns. A node that keeps session data in local memory breaks the moment the load balancer sends the next request to a different node. - Introduces distributed-systems problems for real: network partitions, consistency, coordination, service discovery (
system-design/service_discovery), distributed tracing (system-design/distributed_tracing) just to debug a single user request. - Data layer scaling is a separate, harder problem — see
system-design/database_scaling_strategies_recapanddatabases/scaling.
Session kept in server memory
Request 1 → App 1 (stores cart in RAM)
Request 2 → App 2 (no cart!) → empty basketAdding servers silently breaks users.
Session in a shared store
Request 1 → App 1 → writes cart to Redis
Request 2 → App 2 → reads cart from Redis ✓Any server can handle any request, so servers can be added or removed freely.
4Back-of-Envelope: When Vertical Stops Being Viable
Say a service handles 3,000 req/s today on a 4 vCPU box, and traffic is growing 3x per year.
Year 0: 3,000 req/s → 4 vCPU
Year 1: 9,000 req/s → ~12 vCPU (scale up 3x)
Year 2: 27,000 req/s → ~36 vCPU (scale up 3x again)
Year 3: 81,000 req/s → ~108 vCPU ← at or past the largest single-instance size on most cloudsThree years of 3x growth turns a 4 vCPU box into a fantasy 108 vCPU box that either doesn't exist as a single instance or costs a fortune per hour. Horizontal scaling turns the same growth into "run 27x as many of the same cheap 4 vCPU boxes" — a linear, repeatable operation instead of a wall.
5Hybrid in Practice
Real systems mix both:
- Scale vertically within a node up to a "sweet spot" (the price/performance knee — see Under the Hood), then scale horizontally past that point.
- Stateful components (a primary database) are usually scaled vertically first, then horizontally via replication/sharding (
databases/scaling/replication,databases/scaling/sharding) because they can't trivially be duplicated like a stateless API server. - Stateless components (API servers, web servers) are scaled horizontally almost exclusively once traffic is non-trivial, because horizontal scaling is what a load balancer and auto-scaling group are built to exploit.
stateless API tier stateful data tier
─────────────────── ───────────────────
scale up first? briefly, then scale out yes, much further
scale out easy? yes (add pods/instances) hard (needs sharding/replication)
state? none (external store) the state itself lives hereScale the stateless tier (API servers, workers) horizontally from early on. Scale the stateful tier (the primary database) vertically for as long as it's comfortable, then add read replicas and finally sharding.
Under the HoodThe Real Cost Curve▾
The vertical-scaling cost curve is not linear — it is closer to superlinear (convex) past a certain instance size, for reasons rooted in hardware economics:
relative $/hour vs vCPU count (illustrative, shape matters more than exact numbers):
vCPU: 4 8 16 32 64 96 128
$/hr: 0.16 0.32 0.68 1.54 3.60 6.80 12.50
$/vCPU: 0.040 0.040 0.0425 0.048 0.056 0.071 0.098Why the curve bends upward:
- Binning effects — cloud providers price the largest SKUs in a family at a premium because fewer customers need them and fewer physical hosts can even offer them (you need a single physical machine with that many cores/that much RAM attached).
- Diminishing parallel efficiency — more cores on one machine means more contention on shared resources (memory bus, last-level cache, NUMA node crossings, lock contention in the OS scheduler and in your own application's shared data structures). Doubling cores rarely doubles throughput once you're past ~16–32 cores for most real workloads.
- Horizontal scaling's cost curve is close to linear by contrast: 10 small boxes cost almost exactly 10x one small box, because you're buying more of a commodity SKU that many other customers also want, not renting a rare giant machine.
This is the actual arithmetic reason "scale out" beats "scale up" past a certain size, independent of the fault-tolerance argument: $/req/s gets worse as you scale up, and stays flat as you scale out — until coordination overhead (network hops, cache misses across replicas, load-balancer overhead) eventually makes the horizontal curve bend too, just at much higher N.
There is a second, distinct engineering cost that doesn't show up in a cloud invoice: horizontal scaling only "just works" if the workload decomposes into independent units of work. A CPU-bound single-threaded hot loop (e.g., one giant in-memory graph traversal) parallelizes across cores on one box far more cheaply than it re-architects into a distributed job — which is why vertical scaling remains the right first move for specific compute-bound problems even inside an otherwise horizontally-scaled system.
⚠️ Common Pitfalls
- Scaling out a stateful service without a plan for the state — adding 3 more app servers that each keep an in-memory user session cache means 3/4 of requests hit a "cold" server with no session. Externalize state (Redis, a shared DB) before scaling horizontally, not after.
- Assuming horizontal scaling is free of a ceiling — coordination overhead (chatty service-to-service calls, a shared database that becomes the bottleneck, cross-node cache invalidation) means horizontal scaling has diminishing returns too, just at a much higher N than vertical scaling's hard instance-size wall.
- Vertically scaling a box that's actually I/O-bound, not CPU-bound — throwing more vCPUs at a service that's waiting on disk or network I/O does nothing; profile first (see
system-design/observability_overview) to know which resource is actually saturated. - Ignoring the "thundering herd" on scale-out events — a newly added node with a cold cache can get flooded by the load balancer sending it full-weight traffic immediately; ramp-up/slow-start on the load balancer side (see
system-design/load_balancing_algorithms) avoids this. - Treating scale-up as "buy a bigger box and forget it" — vertical scaling still needs a resize/reboot plan, and a single bigger box is still a single point of failure no amount of RAM fixes.
- Not accounting for licensing costs that scale per-core — some commercial databases and enterprise software license by core count, so vertical scaling can hit a cost wall (not just a hardware wall) well before the biggest instance size.
📋 Quick Reference
Vertical (scale up) Horizontal (scale out)
Unit of change bigger machine more machines
Ceiling hard (largest instance) soft (coordination overhead)
Fault tolerance none (single box) yes (N-1 nodes survive)
Cost curve convex / superlinear late near-linear
App changes needed usually none requires statelessness
Best for stateful primaries, quick fix stateless tiers, elastic load
Downtime to resize often yes (reboot) no (add/remove nodes live)🧠 Key takeaways
- Vertical scaling (bigger machine) is simple but has a hard ceiling, a convex cost curve and no redundancy.
- Horizontal scaling (more machines) is near-linear in cost and survives node failures, but needs stateless services.
- Move state (sessions, caches, files) to shared stores before scaling out.
- Real systems mix both: scale the stateless tier out, scale the database up first, then replicate or shard it.
🎤 Interview Q&A
Q1What's the difference between vertical and horizontal scaling?core▾
Vertical scaling adds resources to one machine (more CPU, RAM, faster disks); horizontal scaling adds more machines and spreads the load. Vertical is simpler (no code changes, no distribution problems) but hits a hardware ceiling, costs grow faster than linearly at the top end, and the box remains a single point of failure. Horizontal scales further and improves availability, but requires stateless services, load balancing and handling distributed data.
- When would you choose vertical scaling first? Early on, or for components that are hard to distribute, like a relational primary: it buys time cheaply until load justifies sharding or replicas.
Q2What does a service need in order to scale horizontally?core▾
It should be stateless: any instance can serve any request. Session state moves to a shared store (Redis, a database) or into tokens; files go to object storage; caches are shared or tolerate misses. You also need a load balancer, health checks, and idempotent request handling so retries on another instance are safe.
- What about sticky sessions? They work but undermine even load distribution and failover; use them only as a stop-gap.
Q3Why is the database usually the hardest tier to scale horizontally?core▾
Stateless app servers can be cloned, but data has to stay consistent. Reads scale with replicas and caches; writes need partitioning (sharding), which complicates joins, transactions, rebalancing and operations. That's why teams typically scale the database vertically and with read replicas for as long as possible.
- What's the first step when the DB becomes the bottleneck? Measure: fix slow queries and indexes, add a cache for hot reads, add read replicas, then consider sharding writes.
Q4How do you know when it's time to scale?core▾
From metrics, not guesses: sustained CPU or memory saturation, rising p95/p99 latency, queue growth, connection pool exhaustion, and error rates as load increases. Load tests tell you the breaking point ahead of time, and growth projections tell you how long your headroom lasts.
- What utilization target do you plan for? Commonly 50–70% at peak, leaving headroom for spikes and the loss of an instance or zone.
Q5How does autoscaling work and what are its pitfalls?senior▾
An autoscaler adds or removes instances based on a metric (CPU, request rate, queue depth) with minimum/maximum bounds and cooldowns. Pitfalls: slow instance start-up means scaling lags spikes; scaling on the wrong metric (CPU when the bottleneck is the DB); downstream systems that can't absorb the extra instances; and flapping without cooldowns.
- How do you handle predictable spikes? Scheduled scaling or pre-warming before the event, plus load shedding as a safety net.
🛠️ Practice exercises
Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.
✅ Check yourself
Answer the questions — feedback is instant.