Courses › System Design › Core Concepts

Load Balancers

L4 vs L7, balancing algorithms, health checks, sticky sessions and avoiding a single point of failure.

As soon as you run more than one copy of a service, something has to decide which copy handles each request. That's the load balancer. It spreads traffic, routes around unhealthy servers, and lets you add or remove capacity without clients noticing. In most designs it's one box on the diagram, but interviewers like to probe what layer it works at, how it picks a server, and what happens to state.

L4 vs L7

[!TIP] L7 is the usual answer For an HTTP API, say "an L7 load balancer" and move on. Mention L4 when you have long-lived connections such as WebSockets, where you want the balancer to forward the TCP stream untouched.

Algorithms

  • Round robin: each server in turn. Fine when requests cost about the same.
  • Weighted round robin: bigger servers get a bigger share.
  • Least connections: send to the server with the fewest open connections. Better for long-lived or uneven requests.
  • Least response time: route to whichever server is answering fastest.
  • IP or key hash: the same client or key always lands on the same server. Use consistent hashing so adding a server only moves a small slice of keys.
  • Power of two choices: pick two servers at random and use the less busy one. Nearly as good as least connections, with far less coordination.

Health checks

The balancer regularly probes each server (for example GET /health every few seconds) and stops sending traffic to one that fails several checks in a row. Two kinds are worth separating:

  • Liveness: the process is running.
  • Readiness: it can actually serve, including its database connections and warm caches.

A server that's alive but not ready should be pulled from rotation, not restarted.

Sticky sessions and state

Sticky sessions pin a client to one server, usually with a cookie. They make in-memory session state work, but they skew load and lose the session when that server dies. The better answer is to make app servers stateless: keep sessions in Redis or a signed token, so any server can handle any request and scaling is just adding servers.

The exception is genuinely stateful connections such as WebSockets. There, the connection itself lives on one server; see Real-time Updates.

Avoiding a single point of failure

A single load balancer is itself a single point of failure. In practice:

  • Run balancers in pairs (active-passive with a floating IP) or as a managed, multi-zone service.
  • Use DNS to spread users across regions, each with its own balancers. DNS-based balancing is coarse, because clients cache answers for the record's TTL.
  • Anycast advertises one IP from many locations so users reach the nearest one. CDNs and global load balancers rely on it.
user = client "Users" @0,1
dns = service "DNS / Anycast" @1,1
lbA = service "L7 LB\n(us-east)" @2,0.5
lbB = service "L7 LB\n(eu-west)" @2,1.5
appA = service "App Servers" @3,0.5
appB = service "App Servers" @3,1.5
user -> dns : nearest region
dns -> lbA, lbB
lbA -> appA
lbB -> appB

Load balancer vs API gateway vs reverse proxy

These overlap, and the same software (NGINX, Envoy) can play every role:

  • A reverse proxy sits in front of servers and forwards requests to them.
  • A load balancer is a reverse proxy whose main job is spreading load across identical servers.
  • An API gateway adds API concerns on top: authentication, rate limiting, request routing to different services. See API Gateways.

[!INTERVIEW] Say where it sits and what it does "Clients hit an L7 load balancer that terminates TLS and spreads requests across stateless API servers with least-connections; sessions live in Redis, so we can autoscale freely" covers placement, algorithm and state in one sentence.