Load balancers: L4 vs L7, hashing, and health checks

Every serious system runs a forwarder whose whole job is sending each request to a backend that can serve it. That forwarder is the load balancer. It must run faster than everything behind it, keep whatever connection state it promises, and notice backend deaths before users do.

This post explains load balancing starting at the packet level. It covers where the balancer sits, what L4 and L7 each touch, how distribution algorithms work, and how health checking and failover keep dead servers out of rotation.

1. Where the balancer sits, and why DNS is not enough

DNS round-robin assigns many A records to one hostname. It is the cheapest balancer, and it has three faults. Clients cache records for the TTL. Dead IPs stay in rotation until the TTL expires. Distribution is chance, with no control over the spread. Real systems put the decision on the request path.

The typical chain:

client → DNS → anycast VIP → L4 balancer → L7 proxy → app servers
              (ECMP spray)   (per-conn)     (per-request)

• Anycast + ECMP spreads packets across balancer machines at the network layer before any LB logic runs. If one LB box dies, routing converges around it.
• L4 balancer picks a backend, or an L7 proxy, per connection using only IP and TCP info. It forwards millions of packets per second with no payload inspection.
• L7 proxy picks per request using HTTP headers, path, and cookies. It understands content and pays for parsing.

Most production setups run both tiers. L4 gives raw scale and connection stability. L7 gives routing intelligence.

2. What L4 and L7 each touch

L4 forwards packets without reading them

A Layer-4 balancer reads only IP addresses and TCP or UDP ports. The TCP handshake terminates on the backend. The balancer rewrites packet headers so the conversation reaches the right machine and replies return to the client. Three designs dominate:

Mode	How it works	Tradeoff
NAT / SNAT	The LB rewrites the destination IP to the backend, and rewrites the source on return	Simple, but the LB sees all traffic in both directions and becomes the bandwidth bottleneck
DSR, or direct server return	The LB rewrites the destination MAC or IP, and the backend replies straight to the client	Replies skip the LB, which keeps LB egress flat, but the backend must own the VIP through a tunnel or a loopback address
Full proxy with TCP splice	The LB terminates client TCP and opens its own TCP connection to the backend	Holds twice the connections, and enables retries and queueing

L4 never sees TLS plaintext or HTTP. It cannot route by URL. It cannot tell requests apart inside a keep-alive connection, so it cannot retry a failed POST on another backend. It cannot shed load by request cost. What it buys is efficiency. Maglev-style L4 designs forward tens of millions of packets per second per box.

L7 terminates TLS and decides per request

A Layer-7 reverse proxy terminates TCP and TLS, parses HTTP, and routes every request separately, including all requests that share one keep-alive connection:

# L7 routes by path and holds per-backend pools
upstream api {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    keepalive 256;
}

server {
    listen 443 ssl;
    location /api/  { proxy_pass http://api; }
    location /ws/   {
        proxy_pass http://ws_pool;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

That inspection enables what L4 cannot do. It covers path and host routing, header-based canaries such as x-experiment: b, gRPC method routing, retries that respect idempotency, per-route timeouts, compression, and WAF filtering.

The price is CPU. TLS handshakes, HTTP parsing, and buffering make L7 roughly an order of magnitude more CPU-hungry per request than L4 forwarding is per packet. Long-lived protocols pin a backend for the life of the connection. WebSocket, gRPC streams, and SSE defeat per-request balancing unless the operator drains connections or forces periodic re-resolution.

3. Distribution algorithms

Round-robin is the default. Each alternative below fixes one of its faults:

• Round-robin and weighted round-robin spread load evenly with zero awareness. One slow backend drags every Nth request. Weights handle unequal capacity but go stale when traffic shape changes.
• Least-connections sends each connection to the backend with the fewest open connections. It works when requests cost about the same and connections stay short. It fails with mixed WebSocket and REST pools, where a WebSocket holder looks busy forever.
• Least-latency with P2C and EWMA tracks each backend's exponentially weighted moving average latency and picks the best of two random choices. It adapts to real slowness instead of connection counts. Finagle, Envoy, and modern sidecars use this family.
• Consistent hashing hashes backends and keys, such as client IDs, cache keys, or shards, onto a ring. Each key sticks to the nearest backend clockwise. Adding or removing a node remaps about 1/N keys instead of everything. Caches and stateful shards need this, because remapping means cold misses.

ring:  [ A ●────── K1,K2 ●────── B ●── K3 ── C ●────── ]
  K1,K2 → A     K3 → B
  add node D between B and C → only some keys near D move

With virtual nodes, each real backend claims 100 or more ring points. Distribution evens out and removal moves few keys. Maglev hashing, from Google's L4, goes further. Every LB box computes the same permutation table, so any box can take over with little per-connection state.

Avoid sticky sessions when you can. Pinning a user to one backend with a cookie fixes statefulness and breaks balance. One hot session overloads a single box while others idle. Prefer shared session stores such as Redis, or consistent hashing on user ID. Both keep affinity and still rebalance when membership changes.

L4 decision

per connection

L7 decision

per request

Reshard cost

~1/N with hashing

4. Health checking

A balancer that keeps a dead backend in rotation is worse than no balancer. Two mechanisms work together.

Active checks send synthetic probes on an interval. A TCP connect or an HTTP GET to /healthz must return 200 within a timeout. Three settings control behavior. They are interval, timeout, and the consecutive-failure threshold, for example max_fails=3 fail_timeout=10s. Aggressive settings eject a healthy node during a GC pause, which causes flapping. Lazy settings feed users errors for a minute.

Passive checks, also called outlier ejection, watch real traffic. If a backend's error rate or latency deviates, for example a 5xx burst or P99 triple the pool's, the balancer ejects it for a sleep window, then lets a trickle of probes back. Ejection catches backends that answer /healthz yet fail real requests.

A solid /healthz has layers:

GET /live: the process runs. Orchestrators use this for restarts.
GET /ready: the process can serve. The DB is reachable, migrations are done, warmup is complete.
GET /deep: dependencies are healthy. This costs more, so the LB polls it rarely.

Point the LB at /ready, not /live. A pod that is alive but still warming its JIT and connection pools must receive no traffic yet.

5. Failover without dropping requests

Detection is half the work. Connection handling is the other half.

1. Draining on deploy. Mark the backend unready. Stop sending new connections. Wait for in-flight requests to finish within a hard deadline, then stop the process. The Kubernetes preStop sleep with terminationGracePeriodSeconds exists for this.
2. Retry budgets, not infinite retries. A failed idempotent GET may retry on another backend. A non-idempotent POST must not replay blindly. Cap retries at a fraction of traffic, for example 20 percent, so one sick backend's retries never become a storm that kills the healthy ones.
3. Fail fast and shed load. When queues fill, return 503 with Retry-After at once instead of queueing until timeouts fire together in a thundering herd. Circuit breakers formalize this. After the failure threshold trips, the breaker short-circuits for the sleep window.
4. Cascading failure drill. Backend A calls backend B. B slows. A's threads pile up waiting. The LB marks A unhealthy and shifts load to the remaining A replicas, which also pile up on B. Per-dependency timeouts, bulkheads with separate pools per dependency, and hedged requests on latency-critical paths stop this spiral.

Putting it together

Decision	Default choice	Reconsider when
Tier	L4 at the edge, L7 per service	A single tier serves fine below about 10 backends
Algorithm	Least-conn at L7, Maglev-style hash at L4	Caches and shards need consistent hashing
Session state	Shared store, no stickiness	Only legacy apps that cannot move state out keep stickiness
Health	Active /ready checks plus passive ejection	A /live check alone misses backends that fail real requests
Deploys	Drain, grace period, retry budget	Rolling restarts without draining drop in-flight requests

Balancer counters record every failure first. Read active endpoints, ejected hosts, retry rate, and per-backend EWMA latency before opening application logs. Most reports of slow apps explain themselves there.