Load Balancers (Network Level)

1Why Load Balancing Exists

A single server handling all traffic for a service is both a capacity ceiling and a single point of failure. A load balancer sits in front of a pool of backend servers, distributes incoming traffic across them, and — just as importantly — stops sending traffic to any backend that's failed a health check, so failures become capacity reductions instead of outages.

This topic covers network-level (Layer 4) load balancing specifically. The HTTP-aware, path/host-routing kind of load balancing (an nginx-style reverse proxy making routing decisions based on the actual HTTP request) is covered in the HTTP proxy/reverse-proxy topic — the two operate at different layers and solve related but distinct problems, detailed in section 2.

2L4 vs L7 Load Balancing

Layer 4 (Transport) load balancing          Layer 7 (Application) load balancing
-----------------------------------          --------------------------------------
Decides based on: IP + port + protocol       Decides based on: the actual HTTP request
  (never reads the HTTP request itself)        (path, headers, cookies, host, method...)
Doesn't need to terminate TLS to route        Often terminates TLS to inspect the request
Forwards raw TCP/UDP packets/segments         Can rewrite/route/cache at the request level
Lower overhead, higher raw throughput          More overhead, but far more routing flexibility
Example: balancing raw TCP connections         Example: /api/* → service A, /images/* → service B
  across a pool with no awareness of what's
  inside them

An L4 load balancer picks a backend once, at connection-setup time, based only on connection-level information (source/dest IP and port, protocol) — it never looks inside the packets. An L7 load balancer (the reverse proxy topic's subject) can make a different routing decision for every single HTTP request, because it actually reads the request.

Real deployments frequently layer both: an L4 load balancer distributing raw traffic across several L7 reverse-proxy instances, which then make the request-aware routing decisions — trading the L4 layer's lower overhead for scale, and the L7 layer's flexibility for correctness.

3Load Balancing Algorithms

Round robin        — backend N, then N+1, then N+2... cycling through the pool in order
Weighted round robin — like round robin, but backends with higher weight get more turns
                        (useful when backends have different capacity)
Least connections   — send the new connection to whichever backend currently has the
                        fewest active connections (adapts to uneven request duration,
                        unlike plain round robin)
IP hash             — hash the client's source IP to consistently pick the same backend
                        for that client (a crude way to get session stickiness without
                        any backend-side session state)

Round robin is the simplest and assumes all requests and all backends are roughly equivalent. Least connections adapts better when request processing time varies significantly (a backend stuck on a slow request won't keep receiving new ones just because "it's its turn"). IP hash trades load-distribution evenness for a simple form of stickiness.

4Health Checks

A load balancer periodically probes each backend — a TCP connection attempt at minimum, or an application-level check (an HTTP request to a /health endpoint, if the load balancer is health-check-aware at that level) — and removes any backend that fails from the active rotation.

Every 5s: LB opens a TCP connection to backend:port
  Success (within timeout) → backend stays IN rotation
  N consecutive failures    → backend marked UNHEALTHY, removed from rotation
  Once healthy again (M consecutive successes) → backend added back

The threshold counts (N failures to mark down, M successes to mark back up) exist specifically to avoid flapping a backend in and out of rotation over a single transient blip — a backend that fails one check due to a momentary GC pause shouldn't be yanked from production traffic instantly, but one that's been failing for 30 seconds straight genuinely should be.

5Sticky Sessions

Some applications keep session state on a specific backend instance (in-memory sessions, for example, rather than a shared session store) — for these, every request from a given client needs to consistently reach the same backend, or the session appears to vanish. This is session affinity / sticky sessions.

At the network level, this is typically achieved via the IP-hash algorithm from section 3 (crude, breaks if the client's apparent IP changes — e.g. behind a shared NAT or a mobile network switching cell towers) — L7 load balancers can do this far more reliably with a cookie the load balancer itself sets and reads, since that's not something an L4-only balancer that never reads the request can do.

The better long-term fix is usually architectural, not load-balancer configuration: moving session state out of individual backend memory and into a shared store (Redis, a database) that any backend can read, removing the need for stickiness entirely.

6Direct Server Return (Brief)

In a standard L4 load-balancing setup, response traffic flows back through the load balancer (client ↔ LB ↔ backend, both directions). Direct Server Return (DSR) is an optimization where the load balancer only handles the inbound request, and the backend sends its response directly back to the client, bypassing the load balancer on the return path.

Standard:  Client <---> LB <---> Backend       (LB handles both directions)
DSR:       Client <---------------- Backend    (response bypasses LB entirely)
           Client ----> LB ----> Backend        (only the request goes through LB)

This matters at very high throughput, where response traffic (often much larger than request traffic — think large file downloads or media streaming) would otherwise all funnel back through the load balancer, making it a bandwidth bottleneck. DSR requires backends to be configured to respond using the load balancer's virtual IP as the source address (so the client doesn't notice anything unusual), which adds real operational complexity — it's a specialized technique used where the bandwidth savings justify it, not a default.

⚠️ Common Pitfalls

1. Confusing L4 and L7 load balancing capabilities — expecting an L4 load balancer to route based on URL path or HTTP headers is a category error; it never looks inside the request at all.
2. Relying on IP-hash stickiness for session affinity without understanding its failure mode — a client whose apparent source IP changes mid-session (shared corporate NAT, mobile carrier IP rotation) loses stickiness silently, appearing to lose their session.
3. Setting health check thresholds too aggressively — marking a backend unhealthy after a single failed check causes unnecessary flapping under normal transient load; too lax a threshold delays removing a genuinely failed backend, extending user-facing errors.
4. Assuming round robin distributes load evenly when request costs vary wildly — round robin balances request count, not actual load; a backend handling several expensive requests in a row via round robin can be overloaded while others idle.
5. Treating sticky sessions as a long-term architecture rather than a stopgap — session affinity papers over the real fix (shared/externalized session state) and reintroduces a single point of failure per session (if that one backend goes down, the sticky client's session is gone).

📋 Quick Reference

L4 (network-level): decides on IP/port/protocol only, never reads the request — lower overhead
L7 (application-level, see reverse-proxy topic): decides on actual HTTP content — more flexible

Algorithms: round robin | weighted round robin | least connections | IP hash

Health checks: periodic probes, N-failures-to-remove / M-successes-to-restore thresholds
  (thresholds exist to avoid flapping on transient blips)

Sticky sessions: same client -> same backend, usually via IP hash (L4) or cookie (L7)
  Better long-term fix: externalize session state instead of relying on stickiness

DSR (Direct Server Return): response bypasses the LB on the way back to the client
  — a throughput optimization for response-heavy traffic, not a default setup

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: net_load_balancers_network_level_theory.md · generated by tools/build_lessons.py