Utilisation and Queueing

Why a server is effectively full long before it reaches 100% busy: waiting time curves upward with utilisation, gently at first and then vertically.

Every point of utilisation you squeeze out costs more latency than the last one. Running at 70% is a choice to buy latency and burst room with a third of the machine.

utilisationheadroomqueueing theoryM/M/1

Try it

Move the dials — the sentence under the picture changes.

Utilisation70%

Service time10 ms

At 70%, a 10 ms request takes 33.3 ms — 23.3 ms of queueing. Past here the curve turns upward fast, which is why the playground’s default headroom target is 70%.

In plain words

Utilisation is how busy something is: a server at 70% is doing work 70% of the time. It feels like a server has 30% left to give. It does not. Requests do not arrive evenly — they bunch — and whenever two arrive at once, one waits. The busier the server, the more often that happens, and the longer the wait. Response time climbs gently up to about 70%, then turns a corner and goes vertical.

The curve

For the simplest model of a queue — random arrivals, one server — the time a request spends in the system is:

service time at 30% busy: 1.4×
at 70%: 3.3×
at 90%: 10×
at 99%: 100×

Drag the dial in the widget. The shape is the point: flat, then a knee, then a wall. Real systems (many servers, non-random arrivals) bend at slightly different places, but every one of them has the wall.

Why the playground says 70%

The headroom target is where on the curve you choose to live. It is a trade:

Run at 50%

safe, costly

Latency barely above service time. Room for a 2× spike. You are paying for a machine that is idle half the time.

Run at 70%

the usual target

Latency ~3× service time — noticeable but fine. Room for a 40% spike, a deploy that takes a copy out, or a slow dependency. The default in the playground and in most SRE practice.

Run at 90%

efficient, fragile

Latency 10× service time and any bump — a 10% traffic rise, one copy restarting — pushes it past 100%, where the queue grows without bound.

The playground turns a part amber at the headroom target and red at 100% for exactly this reason: amber means "the curve has turned", red means "the queue is growing and will not stop".

Losing one copy

Utilisation also decides what a failure does. Four copies at 70% each carry 280% of one copy's capacity. Lose one, and the remaining three carry it at 93% — up the wall. Four copies at 50% lose one and land at 67% — still fine. That is the "losing one instance" line in the playground's inspector, and why N−1 planning is part of headroom.

copies × utilisation = total load        →  after losing one
4      × 70%         = 280%              →  280% / 3 = 93%   (amber → nearly red)
4      × 50%         = 200%              →  200% / 3 = 67%   (fine)

Where it goes wrong

Averages over long windows. "Average CPU 40% today" hides an hour at 95%. Utilisation matters at the busiest minute, not the daily mean.
A queue in front makes it look fine. A part at 110% behind a queue shows no errors — the queue just grows. See message queues: a queue converts overload into delay, it does not remove it.
Autoscaling on utilisation. A CPU metric saturates at 100%, so a part at 300% demand reads 100% and the scaler adds copies a little at a time. The playground's autoscaler offers a demand metric for this reason.
Different resources, different ceilings. CPU at 40% and the connection pool at 100% is a full server. Watch every resource, not the one on the dashboard.

Take this with you

The one idea: a server is full at 70–80%, not 100%. The last 20% of capacity costs latency you cannot afford.
In an interview, state the curve and pick a headroom target on purpose, with the reason.
At work, look at your busiest minute, not your daily average, and check what utilisation you land at with one copy gone.