How to Alert on Your SLO Without Paging for Noise: Write Multi-Window Burn-Rate Alerts in Prometheus and Unit-Test Them With promtool
Learn what an error budget and a burn rate are, write multi-window burn-rate alerts in Prometheus, and prove with promtool unit tests and replayed traffic when each alert fires, then plant nine...
Alerts fail in two opposite ways. Either nobody hears about a real problem until customers complain, or an alert wakes someone at 3 a.m. for a blip that cost almost nothing. In this tutorial you will build alert rules for an HTTP API that avoid both. They fire when the service is spending its monthly allowance of failures too fast, they page a person only when waiting would be expensive, and they open a quiet ticket when the leak is slow. You will also prove how they behave: you will replay five kinds of traffic through the rules, write unit tests that say exactly when each alert should and should not fire, and plant nine mistakes in the rules to check that your tests notice every one.
Table Of Content
- What you will build
- The five ideas you need first
- Why one threshold is not enough
- Prerequisites
- Step 1: Install promtool and check that it runs
- Step 2: Do the error budget arithmetic
- Step 3: Write the alert most people write first
- Step 4: Write the burn-rate alerts
- Step 5: Build a lab that replays traffic through the rules
- Step 6: Replay the traffic and read the results
- What the usual alert does
- What the burn-rate alerts do
- Step 7: Check promtool against an independent model
- Step 8: Turn the replay into unit tests
- Step 9: Break the rules on purpose
- Common mistakes and how to avoid them
- Putting recording rules and alerts in different groups
- Adding for to a page alert without thinking
- A range window with too few samples
- Testing only the loud cases
- A budget typed for the wrong SLO
- Low-traffic services
- Silence when the metric disappears
- Confirm everything works end to end
- Where to go next
- Sources
Nothing here needs a running Prometheus server, an Alertmanager, a Kubernetes cluster or a paid account. The one new tool is promtool, the command line program that ships with Prometheus. You do not need earlier experience with it. If you have written a Prometheus query before you are ready, and if you have not, every rule is explained line by line.
The alert design is the multiwindow, multi-burn-rate alert from chapter 5 of the Google SRE Workbook. What this post adds is the testing: the workbook gives you the rules, and here you check them against traffic you can reproduce on your own machine.
What you will build
- a rule file with three alerts,
SLOBurnFast,SLOBurnSlowandSLOBurnTicket, built on recording rules; - a small Python lab (standard library only) that creates traffic, replays it through
promtooland reports when each alert fires; - an independent Python model of the same alerts, to cross-check
promtool; - a unit test file,
slo_test.yml, that you can run in CI; - a check that nine realistic mistakes in the rule file are all caught by those tests.
The five ideas you need first
Five words carry this whole topic. Read them once and the rest will make sense.
- SLI (service level indicator). A measurement of how well the service is doing. Here it is the share of requests that fail with an HTTP 5xx status code.
- SLO (service level objective). The target for that measurement over a period. Here it is 99.9 percent of requests succeeding over 30 days.
- Error budget. What the SLO lets you get wrong: 100 percent minus 99.9 percent, so 0.1 percent of requests may fail. Spread over 30 days that is the same as 43.2 minutes of total outage.
- Burn rate. How fast you are spending the budget compared with spending it evenly. A burn rate of 1 uses exactly the whole budget in 30 days. A burn rate of 14.4 uses it in about two days.
- Window. The stretch of recent time an alert looks at, for example the last hour. Long windows are steady but slow to notice. Short windows notice quickly but are jumpy.
Two Prometheus words complete the picture. A recording rule saves the result of a query as a new time series, so alerts can reuse it. An alerting rule is a query plus labels and text, and the alert fires while the query returns something.
Why one threshold is not enough
The alert most people write first says “page me when the error ratio over the last five minutes is above the SLO’s failure rate”. It has one speed setting. A tiny blip trips it, a slow leak that will drain the budget in a week trips it just as loudly, and nothing in it tells you how much budget has actually gone.
The workbook’s answer is to alert on the burn rate and to look through two windows at once. The long window says that a meaningful slice of budget has gone. The short window says that the problem is still happening right now, so the alert stops soon after the problem does. The workbook also gives a rule of thumb: “make the short window 1/12 the duration of the long window”. Its Table 5-8, “Recommended parameters for a 99.9% SLO alerting configuration”, is where we start:
| Severity | Long window | Short window | Burn rate | Budget consumed |
|---|---|---|---|---|
| Page | 1 hour | 5 minutes | 14.4 | 2% |
| Page | 6 hours | 30 minutes | 6 | 5% |
| Ticket | 3 days | 6 hours | 1 | 10% |
A page interrupts a person at once. A ticket goes into a queue to be handled in working hours. The three rows are the three alerts you will write.
Prerequisites
- Operating system: tested on Windows 11 with Git Bash. The Linux and macOS archives from the same release page contain the same
promtool, but I did not test them. - Python: 3.13 (tested with 3.13.14). The lab uses only the standard library.
- promtool: from the Prometheus 3.16.0 release (the version this tutorial was tested with). The download is the full Prometheus archive of about 117 MB, and you only extract one program from it.
- Prior knowledge: you know what an HTTP status code is and you have seen a counter such as
http_requests_totalwith acodelabel. If your application names things differently, change the metric and label names in the rule text. - Time: about 40 minutes, most of it reading.
Step 1: Install promtool and check that it runs
Prometheus releases ship promtool inside the same archive as the server. You will download the archive for your platform, check its checksum, and pull out the one program you need. Run these commands in Git Bash (or any POSIX shell):
mkdir slo-lab && cd slo-lab
curl -L -O https://github.com/prometheus/prometheus/releases/download/v3.16.0/prometheus-3.16.0.windows-amd64.zip
curl -L -O https://github.com/prometheus/prometheus/releases/download/v3.16.0/sha256sums.txt
grep "windows-amd64.zip" sha256sums.txt
sha256sum prometheus-3.16.0.windows-amd64.zip
The last two commands print the checksum the release publishes and the checksum of the file you downloaded. They must be identical:
89c03c2f24400fb58668440b6d7226f68564acda27d657e77acfbda42be32cb3 *prometheus-3.16.0.windows-amd64.zip
89c03c2f24400fb58668440b6d7226f68564acda27d657e77acfbda42be32cb3 prometheus-3.16.0.windows-amd64.zip
A matching checksum tells you the download was not corrupted on the way. It does not prove the release page itself is trustworthy, because both numbers come from the same place. Now extract promtool and tell the lab where it is:
python -c "import zipfile; zipfile.ZipFile('prometheus-3.16.0.windows-amd64.zip').extract('prometheus-3.16.0.windows-amd64/promtool.exe', '.')"
export PROMTOOL="$(cygpath -m "$PWD")/prometheus-3.16.0.windows-amd64/promtool.exe"
"$PROMTOOL" --version | head -1
The last command prints the first line of the version output:
promtool, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d)
If you see a version number, you are set. On Windows, cygpath -m turns the Git Bash path into the C:/... form that Python understands. On Linux or macOS, unpack the Linux archive instead and use export PROMTOOL="$PWD/prometheus-3.16.0.linux-amd64/promtool" (I did not test that variant).
Step 2: Do the error budget arithmetic
Before writing any rule, check the numbers in the workbook table yourself. This takes a few lines of Python and removes the mystery from “14.4”. Save this as slo_math.py:
# slo_math.py
"""Error budget and burn-rate arithmetic for a 99.9 percent SLO over 30 days."""
SLO = 0.999
PERIOD_HOURS = 30 * 24
budget = 1 - SLO
print(f"SLO {SLO * 100:g}% over 30 days: error budget {budget * 100:.3f}% of requests,"
f" the same as {budget * PERIOD_HOURS * 60:.1f} minutes of total outage")
print()
print(f"{'burn rate':>9} {'window':>6} {'threshold ratio':>15} {'days to empty':>13} {'budget used':>11}")
for burn, hours in ((14.4, 1), (6, 6), (1, 72)):
days = PERIOD_HOURS / burn / 24
consumed = burn * hours / PERIOD_HOURS
print(f"{burn:>9} {str(hours) + ' h':>6} {burn * budget * 100:>14.4f}% {days:>13.2f} {consumed * 100:>10.1f}%")
Run it with python slo_math.py:
SLO 99.9% over 30 days: error budget 0.100% of requests, the same as 43.2 minutes of total outage
burn rate window threshold ratio days to empty budget used
14.4 1 h 1.4400% 2.08 2.0%
6 6 h 0.6000% 5.00 5.0%
1 72 h 0.1000% 30.00 10.0%
Read the columns from left to right. The threshold ratio is the burn rate times the budget: a burn rate of 14.4 on a 0.1 percent budget means 1.44 percent of requests failing. Days to empty is 30 days divided by the burn rate, so a burn of 14.4 empties the whole budget in about 2.08 days. Budget used is the burn rate times the window length divided by the 720 hours in 30 days, which is the share of the budget used up if the burn lasts the whole window: 2, 5 and 10 percent, the same percentages as the workbook table. That is also where the odd numbers come from. You pick how much budget you are willing to lose before someone is told, 2 percent in an hour for example, and solve for the burn rate: 0.02 times 720 hours, divided by 1 hour, is 14.4.
Step 3: Write the alert most people write first
You will measure the usual alert next to the better ones, so write it down. Create a folder named rules and save this as rules/naive.yml:
groups:
- name: naive
interval: 1m
rules:
- record: job:naive_errors_per_request:ratio_rate5m
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total{job="api"}[5m]))
- record: job:naive_errors_per_request:ratio_rate1h
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[1h]))
/
sum by (job) (rate(http_requests_total{job="api"}[1h]))
- alert: HighErrorRate
expr: job:naive_errors_per_request:ratio_rate5m > 0.001
labels:
severity: page
- alert: BurnFast1hOnly
expr: job:naive_errors_per_request:ratio_rate1h > (14.4 * 0.001)
labels:
severity: page
The file has one rule group that is evaluated every minute. The two record rules turn counters into a ratio. rate(http_requests_total{...}[5m]) is the per-second increase of each counter over the last five minutes, code=~"5.." keeps the series whose status code starts with 5, and sum by (job) adds the series together so that one number per job remains. Dividing failures by all requests gives the share of requests that failed, a number between 0 and 1. The first alert, HighErrorRate, is the common one: page when more than 0.1 percent of requests failed over five minutes. The second, BurnFast1hOnly, uses only the one-hour window at the 14.4 threshold. It is here to show what the short window buys you later.
Step 4: Write the burn-rate alerts
Now the real rules. Save this as rules/slo.yml:
groups:
- name: slo-burn-rate
interval: 1m
rules:
- record: job:slo_errors_per_request:ratio_rate5m
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total{job="api"}[5m]))
- record: job:slo_errors_per_request:ratio_rate30m
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[30m]))
/
sum by (job) (rate(http_requests_total{job="api"}[30m]))
- record: job:slo_errors_per_request:ratio_rate1h
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[1h]))
/
sum by (job) (rate(http_requests_total{job="api"}[1h]))
- record: job:slo_errors_per_request:ratio_rate6h
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[6h]))
/
sum by (job) (rate(http_requests_total{job="api"}[6h]))
- record: job:slo_errors_per_request:ratio_rate3d
expr: |
sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[3d]))
/
sum by (job) (rate(http_requests_total{job="api"}[3d]))
- alert: SLOBurnFast
expr: |
job:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
and
job:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
labels:
severity: page
annotations:
summary: "api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)"
- alert: SLOBurnSlow
expr: |
job:slo_errors_per_request:ratio_rate6h > (6 * 0.001)
and
job:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
labels:
severity: page
annotations:
summary: "api is burning its 30-day error budget at 6x or faster (6h and 30m windows)"
- alert: SLOBurnTicket
expr: |
job:slo_errors_per_request:ratio_rate3d > (1 * 0.001)
and
job:slo_errors_per_request:ratio_rate6h > (1 * 0.001)
labels:
severity: ticket
annotations:
summary: "api is burning its 30-day error budget at 1x or faster (3d and 6h windows)"
Three details are worth understanding before you move on.
- Five ratios, three alerts. The five recording rules compute the failure ratio over 5 minutes, 30 minutes, 1 hour, 6 hours and 3 days. The names follow the workbook’s pattern,
job:slo_errors_per_request:ratio_rate1h. Each alert compares a long-window ratio and a short-window ratio with the same threshold and requires both, joined byand. The thresholds are written as14.4 * 0.001, burn rate times budget, so a reader can see the intent. - Everything is in one group. The Prometheus documentation says “Rules within a group are run sequentially at a regular interval, with the same evaluation time.” Putting the recording rules above the alerts that use them in the same group means each alert sees ratios computed at the same moment.
- Labels and text.
severity: pageorseverity: ticketis what an Alertmanager route would use to decide between waking someone up and filing a ticket. Setting up Alertmanager is outside this tutorial.
Check that both files are valid with "$PROMTOOL" check rules rules/naive.yml rules/slo.yml:
Checking rules/naive.yml
SUCCESS: 4 rules found
Checking rules/slo.yml
SUCCESS: 8 rules found
The count is the number of rules in each file: 2 recording and 2 alerting rules in the first, 5 recording and 3 alerting rules in the second. check rules checks that the files are valid. It cannot tell you whether the thresholds are sensible, and that is what the rest of the tutorial is for.
Step 5: Build a lab that replays traffic through the rules
Real traffic is never the same twice, so to test alerts you need traffic you can repeat. promtool test rules lets you describe counters by hand in a test file and then asks Prometheus’ rule engine what the alerts would have done. The docs explain the notation for the numbers: a+bxn means “series starts at a, then n further samples incrementing by b”. So 0+5999x4319 is a counter that starts at 0 and grows by 5,999 every minute, 4,320 samples in total, which is three days at one sample a minute.
The lab replays five scenarios. Every scenario starts with three days of healthy traffic, so that all windows, even the three-day one, are full when the incident begins, and ends with eight healthy hours so you can watch the alerts reset. Healthy traffic is 6,000 requests a minute (100 a second) with one failing request a minute (code 500, about 0.017 percent). During an incident the failing requests are code 503, which makes the code=~"5.." match matter.
| Scenario | What happens | Why it is in the lab |
|---|---|---|
outage15 |
Every request fails for 15 minutes | A real outage that should page |
blip |
30 percent of requests fail for one minute | A hiccup that should not wake anyone |
burn5x |
0.5 percent fail for two days | A slow leak, about a 5x burn |
burn7x |
0.7 percent fail for 12 hours | A faster leak, about a 7x burn |
oneerror |
One request a minute, and exactly one of them fails | A very quiet service |
Two files do the work. genrules.py writes the two rule files from Steps 3 and 4 (so you can regenerate them) and can also produce the nine broken variants used in Step 9. Save it first and run python genrules.py; it prints wrote rules/slo.yml and rules/naive.yml.
# genrules.py
"""Generate the Prometheus rule files used in this tutorial (standard library only).
python genrules.py writes rules/naive.yml and rules/slo.yml
Mistakes for the planted-mistake check are produced with slo_rules(mut="name").
"""
import pathlib
HERE = pathlib.Path(__file__).resolve().parent
# name, severity, burn rate, long window, short window (Table 5-8 of the Google SRE Workbook, chapter 5)
ALERTS = [
("SLOBurnFast", "page", "14.4", "1h", "5m"),
("SLOBurnSlow", "page", "6", "6h", "30m"),
("SLOBurnTicket", "ticket", "1", "3d", "6h"),
]
WINDOWS = ["5m", "30m", "1h", "6h", "3d"]
SUMMARY = {
"SLOBurnFast": "api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)",
"SLOBurnSlow": "api is burning its 30-day error budget at 6x or faster (6h and 30m windows)",
"SLOBurnTicket": "api is burning its 30-day error budget at 1x or faster (3d and 6h windows)",
}
MUTANTS = [
"factor_low", # 14.4 typed as 1.44
"budget_99", # error budget written for a 99 percent SLO (0.01) instead of 0.001
"or_not_and", # long OR short window instead of AND
"short_eq_long", # short window copy-pasted from the long one
"wrong_severity", # fast page alert labelled as a ticket
"only_500", # errors counted as code="500" instead of code=~"5.."
"rate_5s", # 5m recording rule written with a 5s range
"wrong_cmp", # ticket comparison flipped to <
"for_15m", # fast page alert delayed by for: 15m
]
def ratio(window, err, win_text=None):
w = win_text or window
return (f'sum by (job) (rate(http_requests_total{{job="api",{err}}}[{w}]))\n'
f' /\n'
f' sum by (job) (rate(http_requests_total{{job="api"}}[{w}]))')
def slo_rules(mut=None):
err = 'code="500"' if mut == "only_500" else 'code=~"5.."'
out = ["groups:", " - name: slo-burn-rate", " interval: 1m", " rules:"]
for w in WINDOWS:
wt = "5s" if (mut == "rate_5s" and w == "5m") else None
out += [f" - record: job:slo_errors_per_request:ratio_rate{w}",
" expr: |",
f" {ratio(w, err, wt)}"]
budget = "0.01" if mut == "budget_99" else "0.001"
for name, sev, burn, long_w, short_w in ALERTS:
if name == "SLOBurnFast":
if mut == "factor_low":
burn = "1.44"
if mut == "short_eq_long":
short_w = long_w
if mut == "wrong_severity":
sev = "ticket"
op = "or" if (mut == "or_not_and" and name == "SLOBurnFast") else "and"
cmp_ = "<" if (mut == "wrong_cmp" and name == "SLOBurnTicket") else ">"
out += [f" - alert: {name}",
" expr: |",
f" job:slo_errors_per_request:ratio_rate{long_w} {cmp_} ({burn} * {budget})",
f" {op}",
f" job:slo_errors_per_request:ratio_rate{short_w} {cmp_} ({burn} * {budget})"]
if mut == "for_15m" and name == "SLOBurnFast":
out.append(" for: 15m")
out += [" labels:", f" severity: {sev}",
" annotations:", f' summary: "{SUMMARY[name]}"']
return "\n".join(out) + "\n"
def naive_rules():
out = ["groups:", " - name: naive", " interval: 1m", " rules:"]
for w in ("5m", "1h"):
out += [f" - record: job:naive_errors_per_request:ratio_rate{w}",
" expr: |", f" {ratio(w, 'code=~\"5..\"')}"]
out += [" - alert: HighErrorRate",
" expr: job:naive_errors_per_request:ratio_rate5m > 0.001",
" labels:", " severity: page",
" - alert: BurnFast1hOnly",
" expr: job:naive_errors_per_request:ratio_rate1h > (14.4 * 0.001)",
" labels:", " severity: page"]
return "\n".join(out) + "\n"
if __name__ == "__main__":
(HERE / "rules").mkdir(exist_ok=True)
(HERE / "rules" / "slo.yml").write_text(slo_rules(), encoding="utf-8", newline="\n")
(HERE / "rules" / "naive.yml").write_text(naive_rules(), encoding="utf-8", newline="\n")
print("wrote rules/slo.yml and rules/naive.yml")
The second file, lab.py, is the lab itself. Its main parts are these. build() turns a scenario into three counters, one for each status code we use (200, 500 and 503). compact() writes a list of numbers in the a+bxn notation. model() is an independent Python version of the alert logic, used in Step 7. run_matrix() asks promtool a clever question: for every minute of the replay it writes a check that says “no alert should be firing” (exp_alerts: []). Whenever an alert is actually firing, promtool reports a failed check and prints the alerts it got, so the list of failures is exactly the timeline of when each alert fired. suite_text() and cmd_mutants() belong to Steps 8 and 9.
# lab.py
"""SLO burn-rate alert lab: traffic scenarios, a pure-Python model, promtool runs, planted mistakes.
python lab.py matrix run every scenario through promtool and print when each alert fires
python lab.py suite write slo_test.yml (a readable unit test) and run it on the clean rules
python lab.py mutants run slo_test.yml against nine planted mistakes
Needs promtool (set PROMTOOL=<path> if it is not on PATH). Standard library only.
"""
import json
import os
import pathlib
import re
import shutil
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
import genrules
HERE = pathlib.Path(__file__).resolve().parent
PROMTOOL = os.environ.get("PROMTOOL", "promtool")
WARM = 4320 # three days of healthy history, one sample a minute
TAIL = 480 # eight healthy hours after the incident
SLO_ALERTS = ["SLOBurnFast", "SLOBurnSlow", "SLOBurnTicket"]
ALL_ALERTS = ["HighErrorRate", "BurnFast1hOnly"] + SLO_ALERTS
WINDOW_MIN = {"5m": 5, "30m": 30, "1h": 60, "6h": 360, "3d": 4320}
SCENARIOS = {
# per_min: requests a minute; frac: failing share during the incident; minutes: incident length;
# base: baseline failures a minute (code 500); fixed: exact failures a minute during the incident
"outage15": dict(per_min=6000, frac=1.0, minutes=15, base=1),
"blip": dict(per_min=6000, frac=0.30, minutes=1, base=1),
"burn5x": dict(per_min=6000, frac=0.005, minutes=2880, base=1),
"burn7x": dict(per_min=6000, frac=0.007, minutes=720, base=1),
"oneerror": dict(per_min=1, frac=0.0, minutes=1, base=0, fixed=1),
}
def build(s):
"""Cumulative counters, one sample a minute: sample i holds the traffic of minute i."""
n = WARM + s["minutes"] + TAIL
T = s["per_min"]
ok, e500, e503 = [0] * n, [0] * n, [0] * n
for i in range(1, n):
inc = WARM <= i < WARM + s["minutes"]
x = (s["fixed"] if s.get("fixed") is not None else round(T * s["frac"])) if inc else 0
b = s["base"] if x < T else 0
e503[i] = e503[i - 1] + x
e500[i] = e500[i - 1] + b
ok[i] = ok[i - 1] + T - x - b
return n, ok, e500, e503
def fmt(v):
return str(int(v))
def compact(vals):
"""Turn a list of numbers into promtool's a+bxn notation (n+1 samples starting at a)."""
out, i, n = [], 0, len(vals)
while i < n:
if i + 1 < n:
b = vals[i + 1] - vals[i]
j = i + 1
while j + 1 < n and vals[j + 1] - vals[j] == b:
j += 1
if j - i >= 2:
out.append(f"{fmt(vals[i])}+{fmt(b)}x{j - i}")
i = j + 1
continue
out.append(fmt(vals[i]))
i += 1
return " ".join(out)
def input_block(ok, e500, e503):
lines = []
for code, v in (("200", ok), ("500", e500), ("503", e503)):
lines.append(f" - series: 'http_requests_total{{job=\"api\",code=\"{code}\"}}'")
lines.append(f" values: '{compact(v)}'")
return lines
def ratio_series(n, ok, e500, e503, window):
"""Failing share over the last `window` minutes, with rate()'s left-open range: window - 1 intervals."""
k = window - 1
out = []
for t in range(n):
a = max(0, t - k)
err = (e500[t] + e503[t]) - (e500[a] + e503[a])
tot = (ok[t] + e500[t] + e503[t]) - (ok[a] + e500[a] + e503[a])
out.append(err / tot if tot > 0 else None)
return out
def model(s):
n, ok, e500, e503 = build(s)
R = {w: ratio_series(n, ok, e500, e503, m) for w, m in WINDOW_MIN.items()}
def gt(x, thr):
return x is not None and x > thr
fire = {a: [False] * n for a in ALL_ALERTS}
for t in range(n):
fire["HighErrorRate"][t] = gt(R["5m"][t], 0.001)
fire["BurnFast1hOnly"][t] = gt(R["1h"][t], 14.4 * 0.001)
fire["SLOBurnFast"][t] = gt(R["1h"][t], 14.4 * 0.001) and gt(R["5m"][t], 14.4 * 0.001)
fire["SLOBurnSlow"][t] = gt(R["6h"][t], 6 * 0.001) and gt(R["30m"][t], 6 * 0.001)
fire["SLOBurnTicket"][t] = gt(R["3d"][t], 1 * 0.001) and gt(R["6h"][t], 1 * 0.001)
return fire
DUR = re.compile(r"(\d+)(ms|y|w|d|h|m|s)")
UNIT = {"y": 525600, "w": 10080, "d": 1440, "h": 60, "m": 1, "s": 1 / 60}
def minutes(txt):
return int(round(sum(int(a) * UNIT[u] for a, u in DUR.findall(txt) if u != "ms")))
def run_promtool(args, cwd):
p = subprocess.run([PROMTOOL] + args, cwd=str(cwd), capture_output=True, text=True,
encoding="utf-8", errors="replace", timeout=1500)
return p.returncode, p.stdout + p.stderr
def parse_failures(text):
"""Return [(alertname, minute, fired)] for every failed check in promtool's output."""
res = []
for chunk in re.split(r"alertname: ", text)[1:]:
m = re.match(r"(\w+), time: (\S+?),", chunk)
if not m:
continue
got = chunk.split("got:", 1)[1] if "got:" in chunk else ""
res.append((m.group(1), minutes(m.group(2)), "Labels:" in got))
return res
def run_matrix(name):
s = SCENARIOS[name]
n, ok, e500, e503 = build(s)
d = HERE / "work" / name
d.mkdir(parents=True, exist_ok=True)
out = ["rule_files:", " - ../../rules/naive.yml", " - ../../rules/slo.yml", "evaluation_interval: 1m",
"tests:", " - interval: 1m", " input_series:"] + input_block(ok, e500, e503) + [" alert_rule_test:"]
for t in range(60, n):
for a in ALL_ALERTS:
out += [f" - eval_time: {t}m", f" alertname: {a}", " exp_alerts: []"]
(d / "matrix.yml").write_text("\n".join(out) + "\n", encoding="utf-8", newline="\n")
rc, text = run_promtool(["test", "rules", "matrix.yml"], d)
fired = {a: set() for a in ALL_ALERTS}
for a, t, f in parse_failures(text):
if f:
fired[a].add(t)
return name, rc, fired
def cmd_matrix():
with ThreadPoolExecutor(max_workers=5) as ex:
results = list(ex.map(run_matrix, list(SCENARIOS)))
rows, worst = [], 0
print(f"{'scenario':<9} {'alert':<15} {'first fire':>10} {'budget used':>11} {'minutes':>8} model first fire")
print(f"{'':<9} {'':<15} {'(min in)':>10} {'(percent)':>11} {'firing':>8}")
for name, rc, fired in results:
s = SCENARIOS[name]
n, ok, e500, e503 = build(s)
budget_events = 0.001 * s["per_min"] * 43200
fire = model(s)
for a in ALL_ALERTS:
pt = sorted(fired[a])
mt = [t for t in range(n) if fire[a][t] and t >= 60]
first = pt[0] - WARM if pt else None
mfirst = mt[0] - WARM if mt else None
used = round(100 * e503[pt[0]] / budget_events, 1) if pt else None
diff = (first - mfirst) if (first is not None and mfirst is not None) else None
worst = max(worst, abs(diff or 0))
rows.append(dict(scenario=name, alert=a, first=first, budget_used=used, minutes_firing=len(pt),
model_first=mfirst, model_minutes=len(mt), same_minutes=(set(pt) == set(mt)),
exit_code=rc))
f = "never" if first is None else f"{first:+d}"
u = "-" if used is None else f"{used:.1f}"
mf = "never" if mfirst is None else f"{mfirst:+d}"
print(f"{name:<9} {a:<15} {f:>10} {u:>11} {len(pt):>8} {mf}"
f"{'' if set(pt) == set(mt) else ' (minute sets differ)'}")
print()
same = sum(r["same_minutes"] for r in rows)
print(f"model and promtool fire in exactly the same minutes for {same} of {len(rows)} scenario/alert pairs;"
f" largest first-fire difference {worst} minute(s)")
os.makedirs(HERE / "out", exist_ok=True)
(HERE / "out" / "matrix.json").write_text(json.dumps(rows, indent=1), encoding="utf-8")
SUITE_SCENARIOS = ["outage15", "blip", "burn7x", "oneerror"]
def suite_text(rules_path="rules/slo.yml"):
out = ["rule_files:", f" - {rules_path}", "evaluation_interval: 1m", "tests:"]
for name in SUITE_SCENARIOS:
s = SCENARIOS[name]
n, ok, e500, e503 = build(s)
fire = model(s)
sets = [frozenset(a for a in SLO_ALERTS if fire[a][t]) for t in range(n)]
times = {WARM - 1, WARM, WARM + 1, WARM + 10} # also pin the quiet minutes around the incident start
for t in range(1, n):
if sets[t] != sets[t - 1]:
times.update((t - 1, t))
out += [f" - name: {name}", " interval: 1m", " input_series:"] + input_block(ok, e500, e503)
out.append(" alert_rule_test:")
for t in sorted(times):
for a, sev, _b, _l, _sh in genrules.ALERTS:
out += [f" - eval_time: {t}m", f" alertname: {a}"]
if fire[a][t]:
out += [" exp_alerts:", " - exp_labels:", f" severity: {sev}",
" job: api", " exp_annotations:",
f" summary: \"{genrules.SUMMARY[a]}\""]
else:
out.append(" exp_alerts: []")
return "\n".join(out) + "\n"
def cmd_suite():
(HERE / "rules").mkdir(exist_ok=True)
(HERE / "slo_test.yml").write_text(suite_text(), encoding="utf-8", newline="\n")
rc, text = run_promtool(["test", "rules", "slo_test.yml"], HERE)
print(text.strip()[:1500])
print("exit code", rc)
def cmd_mutants():
text = suite_text()
base = HERE / "mut"
shutil.rmtree(base, ignore_errors=True)
def one(name):
d = base / name
(d / "rules").mkdir(parents=True)
(d / "rules" / "slo.yml").write_text(genrules.slo_rules(None if name == "clean" else name),
encoding="utf-8", newline="\n")
(d / "slo_test.yml").write_text(text, encoding="utf-8", newline="\n")
rc, out = run_promtool(["test", "rules", "slo_test.yml"], d)
os.makedirs(HERE / "out", exist_ok=True)
(HERE / "out" / f"mut_{name}.txt").write_text(out, encoding="utf-8")
return name, rc, parse_failures(out)
names = ["clean"] + genrules.MUTANTS
with ThreadPoolExecutor(max_workers=6) as ex:
results = list(ex.map(one, names))
caught = 0
for name, rc, fails in results:
first = f"{fails[0][0]} at {fails[0][1]}m" if fails else "-"
if name != "clean" and rc != 0:
caught += 1
print(f"{name:<15} exit {rc} failed checks {len(fails):>3} first: {first}")
print(f"planted mistakes caught: {caught} of {len(genrules.MUTANTS)}; clean rules exit code {results[0][1]}")
if __name__ == "__main__":
{"matrix": cmd_matrix, "suite": cmd_suite, "mutants": cmd_mutants}[sys.argv[1]]()
Step 6: Replay the traffic and read the results
Run the replay. All five scenarios run in parallel and take about ten seconds on a machine with 14 logical processors:
python lab.py matrix > matrix.txt
grep -E "^scenario|^ +\(min in\)|HighErrorRate|BurnFast1hOnly" matrix.txt
The columns are: first fire, the minute the alert first fired, counted from the first minute that contains a failing request (+0 is that very minute); budget used, the percentage of the 30-day budget consumed by the incident’s failing requests up to that minute (the healthy background failures are not counted); and minutes firing, how many one-minute evaluations the alert was firing during the replay.
What the usual alert does
scenario alert first fire budget used minutes model first fire
(min in) (percent) firing
outage15 HighErrorRate +0 2.3 18 +0
outage15 BurnFast1hOnly +0 2.3 73 +0
blip HighErrorRate +0 0.7 4 +0
blip BurnFast1hOnly never - 0 never
burn5x HighErrorRate +0 0.0 2883 +0
burn5x BurnFast1hOnly never - 0 never
burn7x HighErrorRate +0 0.0 723 +0
burn7x BurnFast1hOnly never - 0 never
oneerror HighErrorRate +0 2.3 4 +0
oneerror BurnFast1hOnly +0 2.3 59 +0
The five-minute alert (HighErrorRate) fires at the first failing minute in every scenario. For the one-minute blip it pages for four minutes, although the blip used only 0.7 percent of the budget. For the slow leak it pages for 2,883 minutes, the whole two days plus the few minutes the window needs to clear, starting when 0.0 percent of the budget was gone. A person woken up by that page cannot tell from the page whether the budget is going to last six days or six minutes. The one-hour-only alert (BurnFast1hOnly) ignores the blip and the leaks, but it fires for 73 minutes in total for the 15-minute outage, because the hour-long window stays above the threshold long after the service has recovered.
What the burn-rate alerts do
grep -E "^scenario|^ +\(min in\)|SLOBurn|^model and" matrix.txt
scenario alert first fire budget used minutes model first fire
(min in) (percent) firing
outage15 SLOBurnFast +0 2.3 18 +0
outage15 SLOBurnSlow +2 6.9 41 +2
outage15 SLOBurnTicket +3 9.3 370 +3
blip SLOBurnFast never - 0 never
blip SLOBurnSlow never - 0 never
blip SLOBurnTicket never - 0 never
burn5x SLOBurnFast never - 0 never
burn5x SLOBurnSlow never - 0 never
burn5x SLOBurnTicket +719 8.3 2460 +719
burn7x SLOBurnFast never - 0 never
burn7x SLOBurnSlow +299 4.9 425 +299
burn7x SLOBurnTicket +514 8.3 522 +514
oneerror SLOBurnFast +0 2.3 4 +0
oneerror SLOBurnSlow never - 0 never
oneerror SLOBurnTicket never - 0 never
model and promtool fire in exactly the same minutes for 25 of 25 scenario/alert pairs; largest first-fire difference 0 minute(s)
Go through the scenarios one by one:
- outage15. All three alerts fire, one after another.
SLOBurnFastfires in the first failing minute, when a single minute of total outage has already used 2.3 percent of the budget.SLOBurnSlowfollows two minutes later at 6.9 percent, and the ticket three minutes in at 9.3 percent. The fast alert fires for 18 minutes, against 73 for the one-hour-only version: the five-minute window ends the alert about four minutes after the failures stop. - blip. No burn-rate alert fires at all. The usual alert paged for four minutes.
- burn5x. No page. The ticket opens after 719 minutes, about 12 hours, when the incident has used 8.3 percent of the budget. The healthy background failures use roughly the other 1.7 percent, which brings the three-day ratio to the 0.1 percent threshold, the 10 percent row of the workbook table.
- burn7x. A slightly faster leak does page, through the slow alert, after 299 minutes (about five hours) with 4.9 percent of the budget used, which is the 5 percent row of the table. The ticket follows at minute 514.
- oneerror. At one request a minute, a single failed request pages. That is honest arithmetic and not a bug in the rule: at one request a minute the whole month’s budget is only about 43 failed requests, so one failure is 2.3 percent of it. Low-traffic services need a different approach, and the pitfalls section at the end shows the workbook’s options.
Step 7: Check promtool against an independent model
How do you know your expectations of an alert are right? One way is to calculate them a second way. model() in lab.py does the same job in plain Python. For each minute it computes the failure ratio over each window as the failing requests in the window divided by all requests in the window, applies the same thresholds and combines them with the same and.
There is one subtle detail. A window of five minutes does not contain five one-minute intervals. Since Prometheus 3.0 range selectors are left-open: the migration guide says “Lookback and range selectors are now left-open and right-closed (previously left-closed and right-closed)”. With samples exactly one minute apart, a [5m] range holds five samples: the guide says such queries “will always return 5 samples given even spacing”. Five samples are four intervals, so the model uses the window length minus one intervals.
The last line of the Step 6 output compares the two calculations over all five scenarios and all five alerts:
grep "^model and" matrix.txt
It reads: model and promtool fire in exactly the same minutes for 25 of 25 scenario/alert pairs, with a largest difference in first-fire time of 0 minutes. When two independent calculations agree minute for minute, you can trust both the model and the test file generated from it.
Step 8: Turn the replay into unit tests
A replay is an experiment. A unit test is a promise that keeps being checked. lab.py suite writes slo_test.yml, a standard promtool test file, using the model’s answers as the expectations. The beginning of that file looks like this:
rule_files:
- rules/slo.yml
evaluation_interval: 1m
tests:
- name: outage15
interval: 1m
input_series:
- series: 'http_requests_total{job="api",code="200"}'
values: '0+5999x4319 25909681+0x14 25915680+5999x479'
- series: 'http_requests_total{job="api",code="500"}'
values: '0+1x4319 4319+0x14 4320+1x479'
- series: 'http_requests_total{job="api",code="503"}'
values: '0+0x4319 6000+6000x14 90000+0x479'
alert_rule_test:
- eval_time: 4319m
alertname: SLOBurnFast
The three series lines are the counters for the outage15 scenario. Notice how short the notation makes them: 0+5999x4319 is three days of healthy traffic, 25909681+0x14 is the fifteen minutes during which healthy requests stop, and so on. Below the input, each alert_rule_test says “at minute t, the alert X should be firing with these labels and annotations”, or, with exp_alerts: [], “it should not be firing”. The labels you list are the rule’s own labels plus the labels of the series (job), and you leave alertname out. The suite checks every minute at which any alert changes state and the minute before it, plus the minutes around the start of each incident, so it pins down both when alerts fire and when they stay quiet.
Run the suite:
python lab.py suite
Processing test file: slo_test.yml
SUCCESS
exit code 0
promtool test rules exits with 0 when every check passes and 1 when something fails, which is all a CI job needs to know. The suite has 102 checks in four scenarios and takes about 20 seconds.
Step 9: Break the rules on purpose
A test suite that has never failed might be a test suite that cannot fail. So now you will damage the rules nine ways, each a mistake a tired engineer could really make, and check that the suite fails every time. Run python lab.py mutants. It writes a broken copy of rules/slo.yml for each mistake, runs the same slo_test.yml against it and prints one line per mistake:
| Name | What was changed |
|---|---|
factor_low |
The fast alert’s 14.4 typed as 1.44 |
budget_99 |
The budget written as 0.01, as for a 99 percent SLO |
or_not_and |
The fast alert fires when the long window or the short window is high |
short_eq_long |
The fast alert’s short window copy-pasted from the long one, so both are one hour |
wrong_severity |
The fast alert labelled ticket instead of page |
only_500 |
Failures counted as code="500", which misses 503 |
rate_5s |
The five-minute recording rule written with a range of five seconds |
wrong_cmp |
The ticket comparison flipped from > to < |
for_15m |
A for: 15m added to the fast alert to reduce noise |
clean exit 0 failed checks 0 first: -
factor_low exit 1 failed checks 7 first: SLOBurnFast at 4320m
budget_99 exit 1 failed checks 28 first: SLOBurnFast at 4320m
or_not_and exit 1 failed checks 7 first: SLOBurnFast at 4338m
short_eq_long exit 1 failed checks 5 first: SLOBurnFast at 4338m
wrong_severity exit 1 failed checks 9 first: SLOBurnFast at 4320m
only_500 exit 1 failed checks 30 first: SLOBurnFast at 4320m
rate_5s exit 1 failed checks 9 first: SLOBurnFast at 4320m
wrong_cmp exit 1 failed checks 18 first: SLOBurnTicket at 4319m
for_15m exit 1 failed checks 8 first: SLOBurnFast at 4320m
planted mistakes caught: 9 of 9; clean rules exit code 0
The first row is the unchanged rule file, which must pass with exit code 0. Every damaged copy exits with 1, and the line names the first failed check. To see what a failure looks like, here is the beginning of the full promtool output for the for_15m copy:
Processing test file: slo_test.yml
FAILED:
name: outage15,
alertname: SLOBurnFast, time: 3d,
exp:[
0:
Labels:{alertname="SLOBurnFast", job="api", severity="page"}
Annotations:{summary="api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)"}
],
got:[]
name: outage15,
alertname: SLOBurnFast, time: 3d1m,
exp:[
0:
Labels:{alertname="SLOBurnFast", job="api", severity="page"}
Annotations:{summary="api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)"}
],
Read it from the top. name: outage15 is the test group, alertname: SLOBurnFast, time: 3d is the check that failed (3d is minute 4,320, the first failing minute of the outage), exp is what the test expected, and got:[] says the rule produced no alert. With for: 15m the alert stays pending for 15 minutes before it fires, so the check at the first failing minute finds nothing. It is exactly the kind of well-meant change that a green deploy would not reveal.
Common mistakes and how to avoid them
Putting recording rules and alerts in different groups
The documentation promises an order only inside a group: rules run one after another with the same evaluation time. Do not rely on any order between groups, because an alert in one group may then see a ratio that was computed at a different moment. Keep the recording rules above the alerts that use them in a single group, as rules/slo.yml does.
Adding for to a page alert without thinking
The Prometheus docs say “Alerting rules without the for clause will become active on the first evaluation.” The for_15m mistake above shows what a delay costs. Multi-window alerts already have a built-in delay, because the long window must exceed the threshold, so extra for time is usually unnecessary. If your alerts flap, look at keep_firing_for in the alerting rules documentation, which keeps an alert firing for a while after its condition stops being true.
A range window with too few samples
The rate_5s mistake writes [5s] against counters that are sampled once a minute. A five-second range holds at most one sample, and rate() cannot work out a rate from a single sample, so the recording rule produces no output and the alerts that depend on it never fire. Keep every window at least a few scrape intervals long, and let the unit test tell you when you do not.
Testing only the loud cases
Most people write a test for the outage and stop there. The blip scenario is where you learn that the alerts stay quiet, and an over-sensitive mistake such as factor_low can only be caught by a check that expects silence.
A budget typed for the wrong SLO
The budget_99 mistake looks plausible in a code review because 0.01 is a normal-looking number. Set the budget in one place and derive every threshold from it, as genrules.py does, and let slo_math.py show you the resulting percentages.
Low-traffic services
The oneerror scenario paged for one failed request. The workbook lists options for low-traffic services, the first of which is: “Generate artificial traffic to compensate for the lack of signal from real users.” The others are to combine smaller services for monitoring and to change the product so that one failure counts for less. Pick one before trusting a burn-rate alert on a service that handles a few requests an hour.
Silence when the metric disappears
These rules divide failures by all requests. If a service stops exporting its counters altogether, there is nothing to divide, and a query with no data returns no result, so none of the three alerts would fire. This lab did not test that case. Add a separate alert for missing metrics, for example one built on the absent() function, and test it the same way.
Confirm everything works end to end
From the slo-lab folder, these four checks should all pass:
"$PROMTOOL" check rules rules/naive.yml rules/slo.ymlreportsSUCCESSfor both files.python lab.py matrix | tail -1says that the model andpromtoolfire in the same minutes for 25 of 25 pairs.python lab.py suiteprintsSUCCESSand exits with 0.python lab.py mutantsends withplanted mistakes caught: 9 of 9.
To use this in a real repository, keep your rule files and a test file in version control and make CI run promtool check rules followed by promtool test rules. A non-zero exit code from either command should fail the build.
Where to go next
- Route
severity: pageandseverity: ticketin Alertmanager to different receivers, and decide whether a firing page should silence the matching ticket. - Replace the made-up scenarios with the real shape of your traffic, including its daily rhythm, and keep the incident scenarios you have lived through as permanent tests.
- Apply the same burn-rate structure to a latency objective, by counting requests slower than your threshold as the “failures”.
- If your failures come from slow or failing dependencies, our tutorials on HTTP timeouts and deadlines, retries with backoff and jitter and circuit breakers cover the fixes, and the alerts you built here tell you when they are needed.
- For the metric side, see how to turn Prometheus histograms into OTLP metrics.
Sources
- Google SRE Workbook, chapter 5, Alerting on SLOs (Table 5-8, the short-window rule of thumb and the low-traffic options).
- Prometheus documentation: unit testing for rules, alerting rules, recording rules, query functions and the Prometheus 3.0 migration guide.
- Prometheus 3.16.0 release on GitHub, the version used for every output in this post.







No Comment! Be the first one.