SLOs and Error Budgets

In plain words

"Is the site up?" is not a yes/no question at scale — some requests are always failing somewhere. An SLO (service level objective) turns it into a number: "99.9% of requests succeed within 300 ms, measured over 30 days". The gap between that and 100% is the error budget: the failures you have decided are acceptable. Spend it on deploys and experiments. Watch how fast it burns. Page a human only when it is burning fast enough to run out.

Three words, one idea each

The SLI must be what the user experiences — success and speed at the edge — not CPU or queue depth. Those are causes; the SLI is the effect. Keep the SLO tighter than the SLA so you notice before the customer does.

   requests where status < 500 and latency_ms < 300
─────────────────────────────────────────────────────  over the last 30 days
                 all requests

The error budget

of requests may fail at 99.9%

0.1 %

of full outage per month

43 min

at 99.99%

4.3 min

at 99%

7.2 h

The budget makes the reliability-versus-velocity argument a measurement instead of a fight.

Alert on burn rate, not thresholds

The burn rate is how fast you are spending the budget relative to the pace that would use exactly all of it by month's end. 1× means "on track to spend it all". 10× means "gone in three days". 100× means "gone in seven hours".

1. Page on fast burn

   14× over the last hour and over the last 5 minutes: 2% of the month's budget gone in an hour, still happening. Someone must act now.

2. Ticket on slow burn

   1× over three days: not an emergency, but left alone it breaks the SLO. Tomorrow's problem, and a real one.

3. Do not alert on the rest

   A two-minute blip at 0.5× burns nothing that matters. It is noise, and noise is what teaches people to ignore pages.

- alert: ErrorBudgetBurnFast          # page
  expr: >
    (error_ratio_1h  > 14 * 0.001) and
    (error_ratio_5m  > 14 * 0.001)   # 0.001 = 1 − 0.999, the budget as a ratio
  labels: { severity: page }

- alert: ErrorBudgetBurnSlow          # ticket
  expr: >
    (error_ratio_3d  > 1 * 0.001) and
    (error_ratio_6h  > 1 * 0.001)
  labels: { severity: ticket }

The two windows in each rule (long and short) make sure the problem is both significant and still happening when the alert fires.

What to watch, then

For each service, four signals cover most of it — latency, traffic, errors, saturation. Latency and errors feed the SLI. Traffic explains changes. Saturation — how full the thing is, which the playground reports as utilisation and slots — is the leading indicator: it goes up before errors do, which is what utilisation and queueing is about.

Where it goes wrong

• SLIs on causes. "CPU > 80%" pages someone about a machine that is serving fine. Measure what users see; treat CPU as a clue.
• Too many nines. A 99.99% objective on an internal tool means a four-minute monthly budget, constant pages, and no room to ship. Pick the number the users need, and no tighter.
• Threshold alerts. "Error rate > 1%" is both too noisy (blips) and too quiet (0.9% forever). Burn rate handles both.
• A budget nobody reads. If the number is not on the wall, it does not change what gets shipped. Put it in the deploy tooling: over budget, the button asks why.

Take this with you

• The one idea: an SLO turns "up" into a number; the error budget turns the gap into something you can spend; burn rate turns it into an alert that is worth waking for.
• In an interview, define the SLI on what users see, give an SLO with a window, and say you would alert on burn rate.
• At work, count last month's pages and ask how many mattered. If most did not, the alerts are on thresholds — move them to burn rate.