Load Balancing

Spreading requests across a fleet. The strategies only diverge once requests stop costing the same.

Distributing requests evenly is not the same as distributing work evenly.

round robinleast connectionsweighted

Try it

Move the dials — the sentence under the picture changes.

Strategy

Request cost spreadidentical

With every request costing the same, all four strategies land in the same place — the choice is invisible. Raise the cost spread to make requests unequal, which is what real traffic looks like.

What it is

A load balancer sits in front of a pool of identical servers and decides which one handles each request. That is the whole job. The interesting part is the word "decides", because the strategies only differ under conditions the simple demos never show.

The strategies

Round robin hands out requests in turn. It is stateless, requires nothing from the servers, and distributes requests perfectly evenly. It has no idea whether a request is a 2 ms cache hit or a 4-second report.

Least connections routes to whichever server currently holds the fewest open connections. This is the only common strategy that reacts to what is actually happening — a server that drew several expensive requests naturally stops receiving new ones until it catches up. It needs live state about every backend.

Weighted round robin lets you say that server A is twice the machine server D is. It fixes mixed hardware, not mixed requests.

Random is round robin without the counter. With enough requests the distribution converges to the same place; with few requests it is lumpier. Its real advantage is that it needs no shared state at all, which matters when the load balancer itself is a fleet.

Layer 4 vs layer 7

An L4 (transport layer) balancer forwards TCP connections without reading them: fast, cheap, and blind to what the request contains. An L7 (application layer) balancer is the endpoint the client actually talks to: it reads the HTTP request, so it can route by path or header, retry idempotent requests (ones that are safe to repeat), and split traffic by content — at the cost of doing real work per request and needing to hold the TLS certificates.

Health checks are the real feature

A load balancer's most valuable behaviour is not distribution, it is removing a sick backend from the pool. That makes the check definition the important decision: a check that only pings a port keeps routing traffic to a process whose database connection died, and a check that exercises the whole dependency graph will pull the entire fleet out when one shared dependency blips.

Where it goes wrong

Sticky sessions pin a user to one backend, which makes that backend's death a user-visible event and makes the pool impossible to drain (move traffic off gracefully) evenly. Prefer moving session state out.
Thundering herd on recovery. A backend that rejoins the pool with an empty cache can receive its full share immediately and fall straight back over. Slow start ramps it up.
The balancer is now the single point of failure, which is why the real answer usually involves DNS or anycast (one address announced from several places, so the network routes each client to the nearest) in front of several of them.

Take this with you

The one idea: spreading requests is the easy part. Health checks and how a recovering server rejoins are what decide whether a bad hour becomes an outage.
In an interview, name round robin vs least connections and say when the difference shows (uneven requests). Mention L4 vs L7 and what a health check should test.
At work, read your health check. If it only opens a port, a server with a dead database connection still gets traffic.