How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
Run a real Alertmanager 0.35.0 on your own machine, build a routing tree that sends pages and tickets to the right receivers, measure grouping, timers, inhibition, silences and muted time intervals,...
Prometheus decides when an alert fires. It does not decide who hears about it. That second job belongs to Alertmanager, and it is where many monitoring setups quietly go wrong. A page that should have woken the on-call engineer lands in a ticket queue nobody reads until Monday. A harmless warning wakes someone at 3 a.m. because a label was spelled differently. One broken database produces forty separate messages, and the one that matters is buried among them.
Table Of Content
- What you will build
- The words you need first
- Prerequisites
- Step 1: Download Alertmanager and check the download
- Step 2: Send a first alert and read what arrives
- Step 3: Build the routing tree and test it without sending anything
- Step 4: Know which labels your rules really produce
- Step 5: Control how many notifications you get with group_by
- Step 6: Understand the three timers by watching a clock
- Step 7: Stop tickets from piling on a page with inhibition
- Step 8: Mute alerts on purpose with silences
- Step 9: Keep tickets quiet outside working hours with time intervals
- Step 10: Put the routing tree under a CI gate and test the gate itself
- Common mistakes, in one place
- Confirm that everything works end to end
- Where to go next
- Sources and further reading
In this tutorial you will run a real Alertmanager on your own machine, send it alerts, and watch what it does with them. You will build a routing tree that sends pages to an on-call receiver, sends payment problems to the payments team, and puts tickets in a queue. You will learn what grouping, the three timers, inhibition, silences and time-based muting actually do, with measurements instead of guesses. Then you will write a small check that runs in CI, plant fourteen mistakes in the configuration, and see which ones the check catches and which three it cannot see.
Nothing here needs a Prometheus server, a Kubernetes cluster, a cloud account or a paid service. A short Python program plays the part of PagerDuty, Slack or a ticket system. The alerts in this post carry the same severity labels, page and ticket, as the burn-rate alerts in the earlier post How to Alert on Your SLO Without Paging for Noise, so you can treat this one as the next stage of the same pipeline: that post makes the alerts fire at the right time, and this one gets them to the right place.
What you will build
- a routing tree in
routes.ymlwith four receivers:oncall,payments-oncall,ticketsand a catch-all calledtriage; - a webhook receiver and a small harness that start a real Alertmanager 0.35.0, post alerts to it and print what arrives, with a timestamp;
- one experiment each for grouping, the three timers, inhibition, silences and muted time intervals;
- a CI gate,
gate.py, that fails when an alert would go to the wrong receiver, plus a mutation check that plants fourteen mistakes to measure what the gate can and cannot see.
The words you need first
Alertmanager has its own small vocabulary. Read these once and the rest of the post will make sense.
- Alert. A record that Prometheus (or any program) sends to Alertmanager. It has labels, which are name and value pairs such as
severity=page, optional annotations, which are free text such as a summary, and a start time. The set of labels is the alert’s identity: two alerts with the same labels are the same alert. - Receiver. A named destination, such as a pager, a chat channel or a webhook URL. Alertmanager ships integrations for many services; this post uses the generic webhook one.
- Routing tree. A tree of routes. Each route can have matchers (conditions on labels such as
severity="page"), a receiver, timers, and child routes. Every alert enters at the root and is handed to the receiver of the deepest route that matches it. - Group. Alerts that share a route and the same values for the labels listed in
group_by. Alertmanager sends one notification per group, not one per alert. - Inhibition. A rule that mutes some alerts while another alert is firing, for example “do not send tickets about a service while a page about the same service is open”.
- Silence. A temporary mute that you create by hand, for example during planned maintenance.
- Notification. What a receiver gets: one message for one group, listing every firing and recently resolved alert in it.
The configuration reference puts the first idea this way: “Grouping categorizes alerts of similar nature into a single notification.” It describes inhibition as “suppressing notifications for certain alerts if certain other alerts are already firing” and silences as “a straightforward way to simply mute alerts for a given time”. Those three sentences are the whole feature set you will be testing.
Prerequisites
- Operating system: tested on Windows 11 with Git Bash. The same releases have Linux and macOS archives. I did not test those, and the only change the lab needs is dropping
.exefrom two names inamlab.py. - Alertmanager and amtool: version 0.35.0, released on November 9, 2026. The download is one archive of about 44 MB that holds both programs.
- Python: 3.13 (tested with 3.13.14). The lab uses only the standard library.
- promtool 3.16.0: only for step 4. The earlier SLO post shows how to install it, and you can skip step 4 without losing anything else.
- Prior knowledge: you can read YAML and you know that a metric has labels. You do not need to have run Alertmanager before.
- Time: about 45 minutes, most of it reading. The longest experiment runs 44 seconds.
Step 1: Download Alertmanager and check the download
Alertmanager and its command line tool, amtool, are single programs with no installer. You will download the archive, compare its checksum with the one the project publishes, and copy the two programs into a bin folder. Run this in Git Bash (or any POSIX shell):
mkdir am-lab && cd am-lab
curl -L -o am.zip https://github.com/prometheus/alertmanager/releases/download/v0.35.0/alertmanager-0.35.0.windows-amd64.zip
curl -L -o sha256sums.txt https://github.com/prometheus/alertmanager/releases/download/v0.35.0/sha256sums.txt
grep "windows-amd64.zip" sha256sums.txt
sha256sum am.zip
python -m zipfile -e am.zip unpacked
mkdir bin
cp unpacked/alertmanager-0.35.0.windows-amd64/alertmanager.exe unpacked/alertmanager-0.35.0.windows-amd64/amtool.exe bin/
bin/alertmanager.exe --version 2>&1 | head -1
bin/amtool.exe --version 2>&1 | head -1
The two checksum lines must be identical. The version lines should look like this:
cdf8cd664889bf3f27771881e10814f3d5916310aba90717b4424deb358b0fa3 alertmanager-0.35.0.windows-amd64.zip
cdf8cd664889bf3f27771881e10814f3d5916310aba90717b4424deb358b0fa3 *am.zip
alertmanager, version 0.35.0 (branch: HEAD, revision: 8cf67a8e75004bad6c3c6bacc829945cf8a3ee65)
amtool, version 0.35.0 (branch: HEAD, revision: 8cf67a8e75004bad6c3c6bacc829945cf8a3ee65)
If the checksums differ, delete the file and download it again; do not run a program whose checksum does not match. For Linux or macOS pick the archive for your platform from the v0.35.0 release page and compare against the same sha256sums.txt.
Step 2: Send a first alert and read what arrives
Before the routing tree, you need to see the whole round trip once: an alert goes in, a notification comes out. Alertmanager’s generic webhook receiver POSTs a JSON document to a URL. So the lab needs something that listens on a URL, remembers what arrives and prints it. That is receiver.py, about 50 lines of standard library Python:
# receiver.py
"""receiver.py: a stand-in for PagerDuty, Slack or a ticket system.
It accepts Alertmanager webhook POSTs, remembers them, and prints one block per
notification. Run it on its own with: python receiver.py 9099
"""
import json
import sys
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
def short(labels):
return " ".join(f"{k}={v}" for k, v in sorted(labels.items()))
def render(t, name, payload):
head = f"{t:5.1f}s {name:<16} {payload['status']:<8} group={{{short(payload['groupLabels'])}}}"
lines = [head]
for a in payload["alerts"]:
lines.append(f" {a['status'][:1].upper()} {short(a['labels'])}")
return "\n".join(lines)
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
size = int(self.headers.get("Content-Length", 0))
payload = json.loads(self.rfile.read(size))
name = self.path.strip("/")
self.server.log.append((time.monotonic(), name, payload))
self.send_response(200)
self.end_headers()
if self.server.echo:
print(render(time.monotonic() - self.server.t0, name, payload), flush=True)
def log_message(self, *args):
pass
class Sink(ThreadingHTTPServer):
daemon_threads = True
def __init__(self, port, echo=False):
super().__init__(("127.0.0.1", port), Handler)
self.log = []
self.echo = echo
self.t0 = time.monotonic()
if __name__ == "__main__":
port = int(sys.argv[1]) if len(sys.argv) > 1 else 9099
print(f"listening on 127.0.0.1:{port}", flush=True)
Sink(port, echo=True).serve_forever()
The Sink class is a normal HTTP server that stores every POST body together with a clock reading. render turns one notification into a readable block: the time since the start, the receiver’s name, whether the group is firing or resolved, the group’s labels, and then one line per alert with F for firing or R for resolved.
The second helper, amlab.py, is the harness every later step uses. It writes a configuration into its own folder, starts a real alertmanager.exe on its own port, starts a Sink for the webhooks, and offers four calls: post to send alerts, resolve to end them, alerts to read Alertmanager’s own view, and notifications to read what the sink received. It also holds the table that shrinks the timers, which you will meet in step 6. Save it next to receiver.py:
# amlab.py
"""amlab.py: start a real Alertmanager next to a webhook sink, then talk to both.
Windows, Python 3.13, standard library only. Alertmanager 0.35.0 lives in bin/.
"""
import json
import os
import re
import shutil
import subprocess
import threading
import time
import urllib.request
from datetime import datetime, timedelta, timezone
from receiver import Sink, render
HERE = os.path.dirname(os.path.abspath(__file__)).replace("\\", "/")
AM = f"{HERE}/bin/alertmanager.exe"
AMTOOL = f"{HERE}/bin/amtool.exe"
N = int(os.environ.get("LAB_N", "0")) # lets several scripts run side by side on different ports
# The lab shrinks every timer so that minutes become seconds.
FAST = [
("group_wait: 30s", "group_wait: 2s"),
("group_wait: 10s", "group_wait: 1s"),
("group_interval: 5m", "group_interval: 5s"),
("repeat_interval: 4h", "repeat_interval: 60s"),
("repeat_interval: 1h", "repeat_interval: 30s"),
("repeat_interval: 24h", "repeat_interval: 60s"),
]
def read(name):
with open(f"{HERE}/{name}", encoding="utf-8") as f:
return f.read()
def fast(text):
for old, new in FAST:
assert old in text, old
text = text.replace(old, new)
return text
def amtool(*args, cwd=HERE, env=None):
p = subprocess.run([AMTOOL, *args], capture_output=True, text=True, encoding="utf-8", cwd=cwd,
env=dict(os.environ, **env) if env else None)
return (p.stdout + p.stderr).strip(), p.returncode
def mask(text):
return re.sub(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", "<id>", text)
def stamp(dt):
return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def state_line(a):
s = a["status"]
why = [name for name, key in (("inhibited", "inhibitedBy"), ("silenced", "silencedBy"), ("muted", "mutedBy")) if s.get(key)]
labels = " ".join(f"{k}={v}" for k, v in sorted(a["labels"].items()) if k != "alertname")
return f"{a['labels']['alertname']:<13}{labels:<42}{s['state']:<11}{'+'.join(why)}"
class Lab:
def __init__(self, name, config, k=0, ports=None, echo=False):
self.name = name
self.am_port, self.rx_port = ports or (9100 + 10 * N + k, 9200 + 10 * N + k)
self.dir = f"{HERE}/work/{name}"
shutil.rmtree(self.dir, ignore_errors=True)
os.makedirs(self.dir)
self.cfg = f"{self.dir}/alertmanager.yml"
with open(self.cfg, "w", newline="\n", encoding="utf-8") as f:
f.write(config.replace("9099", str(self.rx_port)))
self.url = f"http://127.0.0.1:{self.am_port}"
self.sink = Sink(self.rx_port, echo=echo)
threading.Thread(target=self.sink.serve_forever, daemon=True).start()
self.proc = None
def start(self):
self.out = open(f"{self.dir}/alertmanager.log", "w")
self.proc = subprocess.Popen(
[AM, f"--config.file={self.cfg}", f"--storage.path={self.dir}/data",
f"--web.listen-address=127.0.0.1:{self.am_port}", f"--web.external-url=http://localhost:{self.am_port}",
"--cluster.listen-address="],
stdout=self.out, stderr=subprocess.STDOUT)
for _ in range(150):
try:
urllib.request.urlopen(self.url + "/-/ready", timeout=1)
break
except Exception:
time.sleep(0.1)
self.t0 = time.monotonic()
self.sink.t0 = self.t0
return self
def now(self):
return time.monotonic() - self.t0
def wait_until(self, t):
time.sleep(max(0.0, t - self.now()))
def post(self, *label_sets, started=None, ended=None):
now = datetime.now(timezone.utc)
body = []
for labels in label_sets:
alert = {"labels": labels, "annotations": {"summary": labels["alertname"]},
"startsAt": stamp(started or now)}
if ended:
alert["endsAt"] = stamp(ended)
body.append(alert)
req = urllib.request.Request(self.url + "/api/v2/alerts", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
urllib.request.urlopen(req, timeout=5).read()
def resolve(self, *label_sets):
now = datetime.now(timezone.utc)
self.post(*label_sets, started=now - timedelta(seconds=1), ended=now)
def alerts(self):
return json.load(urllib.request.urlopen(self.url + "/api/v2/alerts", timeout=5))
def notifications(self):
return [(t - self.t0, name, p) for t, name, p in self.sink.log]
def stop(self):
if self.proc:
self.proc.terminate()
self.proc.wait(timeout=10)
self.sink.shutdown()
self.sink.server_close()
self.out.close()
def print_notifications(lab):
for t, name, p in lab.notifications():
print(render(t, name, p))
Two details are worth knowing now. The harness starts Alertmanager with --cluster.listen-address= and an empty value. The project README says of that flag: “empty string disables HA mode”, and a lone test instance does not need the clustering that high availability uses. And it passes --web.external-url, because without it Alertmanager builds the address it puts in notifications from the machine’s host name, and my first test run printed mine. Now the smallest useful configuration:
route:
receiver: sink
group_wait: 2s
receivers:
- name: sink
webhook_configs:
- url: http://127.0.0.1:9099/sink
One route (the root, which every config needs) with one receiver, and a group_wait of two seconds so you do not wait the default thirty. The first script starts the sink and Alertmanager on the real default ports, adds an alert with amtool, waits five seconds and prints the notification:
# s2_first.py
"""s2_first.py: start Alertmanager with first.yml, add one alert with amtool, show the webhook payload."""
import json
import subprocess
import time
from amlab import AMTOOL, Lab, read
lab = Lab("first", read("first.yml"), ports=(9093, 9099), echo=True).start()
cmd = [AMTOOL, "alert", "add", "alertname=DiskFull", "service=checkout", "severity=page",
'--annotation=summary="Disk is 95 percent full"', "--alertmanager.url=http://127.0.0.1:9093"]
print("$ amtool alert add alertname=DiskFull service=checkout severity=page --annotation='summary=\"Disk is 95 percent full\"' --alertmanager.url=http://127.0.0.1:9093")
p = subprocess.run(cmd, capture_output=True, text=True)
print("exit code", p.returncode, (p.stdout + p.stderr).strip())
time.sleep(5)
t, name, body = lab.notifications()[0]
print(f"the first webhook call arrived {t:.1f} s after the alert was added, at /{name}")
print(json.dumps(body, indent=2))
lab.stop()
Run it with python s2_first.py. Here is what it printed:
$ amtool alert add alertname=DiskFull service=checkout severity=page --annotation='summary="Disk is 95 percent full"' --alertmanager.url=http://127.0.0.1:9093
exit code 0
2.0s sink firing group={}
F alertname=DiskFull service=checkout severity=page
the first webhook call arrived 2.0 s after the alert was added, at /sink
The first block, with F in it, is receiver.py doing its job as the notification arrives, two seconds after the alert was added. That delay is the group_wait you configured. Here is the JSON itself:
{
"receiver": "sink",
"status": "firing",
"alerts": [
{
"status": "firing",
"labels": {
"alertname": "DiskFull",
"service": "checkout",
"severity": "page"
},
"annotations": {
"summary": "Disk is 95 percent full"
},
"startsAt": "2026-11-15T15:08:35.571244-05:00",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "",
"fingerprint": "b0cf240b74b4dde1"
}
],
"notification_reason": "first notification",
"groupLabels": {},
"commonLabels": {
"alertname": "DiskFull",
"service": "checkout",
"severity": "page"
},
"commonAnnotations": {
"summary": "Disk is 95 percent full"
},
"routeLabels": {},
"externalURL": "http://localhost:9093",
"version": "4",
"groupKey": "{}:{}",
"truncatedAlerts": 0
}
This is the payload the configuration reference documents under webhook_config, which it introduces with “The Alertmanager will send HTTP POST requests in the following JSON format to the configured endpoint”. Read it from the top. version is the payload format and is "4". status is firing because at least one alert in the group is firing. alerts lists every alert, each with its own status, its labels and annotations, a startsAt and an endsAt. An endsAt of 0001-01-01T00:00:00Z means “not ended”. groupLabels is empty because this configuration has no group_by. commonLabels holds the labels every alert in the group shares. groupKey identifies the group, and notification_reason says why this message was sent; here it is the first one. The fingerprint is the alert’s identity, computed from its labels.
A gotcha with the annotation. My first attempt passed the annotation without quotes, as --annotation=summary=Disk is 95 percent full. The alert was still added and the exit code was 0, but amtool printed a warning that starts “Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible”, and suggested the quoted form summary="Disk is 95 percent full". The configuration reference says the same about the new parser: “make sure all regular expressions and values are double-quoted”. Quote every value that holds spaces or punctuation, and in a POSIX shell wrap the whole argument in single quotes so the inner double quotes survive, as the script above does.
How to know it worked: the sink printed one firing block with your alert about two seconds after the amtool alert add line. If nothing arrives, check that port 9099 is free and that alertmanager.exe started; the harness writes its log to work/first/alertmanager.log.
Step 3: Build the routing tree and test it without sending anything
Now the real configuration. The goal: pages go to the on-call receiver, pages about payments go to the payments team’s own on-call, tickets go to a queue, and anything that does not match is caught by a receiver called triage where a human can look at it and fix the labels. Here is routes.yml:
# routes.yml: pages go to the on-call phone, tickets go to a queue,
# and anything that matches neither lands in triage.
route:
receiver: triage
group_by: [alertname, service]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [severity="page"]
receiver: oncall
group_wait: 10s
repeat_interval: 1h
routes:
- matchers: [team=~"payments|billing"]
receiver: payments-oncall
- matchers: [severity="ticket"]
receiver: tickets
repeat_interval: 24h
inhibit_rules:
- name: page-covers-ticket
source_matchers: [severity="page"]
target_matchers: [severity="ticket"]
equal: [service]
receivers:
- name: triage
webhook_configs:
- url: http://127.0.0.1:9099/triage
- name: oncall
webhook_configs:
- url: http://127.0.0.1:9099/oncall
- name: payments-oncall
webhook_configs:
- url: http://127.0.0.1:9099/payments-oncall
- name: tickets
webhook_configs:
- url: http://127.0.0.1:9099/tickets
Read the route block from the top. The root route has no matchers and names triage as its receiver. The reference says the top-level route “must match all alerts (i.e. not have any configured matchers)”, so anything that matches no child ends up at the root’s own receiver. It also sets group_by: [alertname, service] and three timers. Child routes inherit these: “Most optional configuration parameters (receiver, group_by, group_wait, group_interval, repeat_interval) are inherited from its parent node if not set.” The page route overrides two timers, so pages wait 10 seconds instead of 30 and repeat every hour instead of every four.
Inside the page route sits a child that matches team=~"payments|billing". The =~ operator means “matches this regular expression”. A page about payments takes that child and goes to payments-oncall; any other page stays at oncall. The reference describes the walk: “If continue is set to false, it stops after the first matching child”, and “If an alert does not match any children of a node (no matching child nodes, or none exist), the alert is handled based on the configuration parameters of the current node.”
The inhibit_rules block is explained in step 7. The receivers at the end all POST to the sink on port 9099, each under its own path so you can tell them apart. Now ask amtool to check the file and draw the tree. Neither command needs a running Alertmanager, because they only read the file:
# s3_routes.py
"""s3_routes.py: read the routing tree and ask amtool where a handful of alerts would go."""
import sys
from amlab import amtool
sys.stdout.reconfigure(encoding="utf-8")
def show(cmd):
out, rc = amtool(*cmd)
print("$ amtool " + " ".join(cmd))
print(out)
print()
show(["check-config", "routes.yml"])
show(["config", "routes", "show", "--config.file=routes.yml"])
CASES = [
"alertname=PaymentsErrors severity=page service=checkout team=payments",
"alertname=DiskFull severity=page service=search team=search",
"alertname=SlowQueries severity=ticket service=search",
"alertname=DiskFull severity=critical service=search",
"alertname=DiskFull severity=Page service=search",
"alertname=NodeDown",
"alertname=PaymentsErrors severity=page service=checkout team=payments-eu",
]
for labels in CASES:
show(["config", "routes", "test", "--config.file=routes.yml", *labels.split()])
show(["config", "routes", "test", "--tree", "--config.file=routes.yml", "severity=page", "team=billing"])
show(["config", "routes", "test", "--config.file=continue.yml", "severity=page"])
show(["config", "routes", "test", "--config.file=routes.yml", "--verify.receivers=tickets", "severity=page"])
Run python s3_routes.py. It prints a lot, so I will take the output in pieces. First the syntax check and the tree:
$ amtool check-config routes.yml
Checking 'routes.yml' SUCCESS
Found:
- global config
- route
- 1 inhibit rules
- 4 receivers
- 0 templates
$ amtool config routes show --config.file=routes.yml
Routing tree:
.
└── default-route receiver: triage
├── {severity="page"} receiver: oncall
│ └── {team=~"payments|billing"} receiver: payments-oncall
└── {severity="ticket"} receiver: tickets
The tree is the quickest way to spot a mistake, because it shows the nesting you intended and the matchers exactly as Alertmanager parsed them. Now seven alerts, each as a command that asks “where would you send an alert with these labels?”:
$ amtool config routes test --config.file=routes.yml alertname=PaymentsErrors severity=page service=checkout team=payments
payments-oncall
$ amtool config routes test --config.file=routes.yml alertname=DiskFull severity=page service=search team=search
oncall
$ amtool config routes test --config.file=routes.yml alertname=SlowQueries severity=ticket service=search
tickets
$ amtool config routes test --config.file=routes.yml alertname=DiskFull severity=critical service=search
triage
$ amtool config routes test --config.file=routes.yml alertname=DiskFull severity=Page service=search
triage
$ amtool config routes test --config.file=routes.yml alertname=NodeDown
triage
$ amtool config routes test --config.file=routes.yml alertname=PaymentsErrors severity=page service=checkout team=payments-eu
oncall
Take these one at a time, because each teaches something.
- Cases 1 to 3 do what you meant: a payments page goes to
payments-oncall, a page from another team tooncall, a ticket totickets. - Case 4 used
severity=critical, a word that exists in many companies’ conventions but not in this tree, and it went totriage. Case 5 isseverity=Page, with a capital letter, and it went totriagetoo: matching is case sensitive. Case 6 has noseveritylabel at all. These three are the silent failures. Nothing errors. The alert is delivered, just not to the place you wanted, and a real page ends up in a channel built for leftovers. This is why the catch-all receiver should be somewhere people actually look. - Case 7 used
team=payments-eu, and it went tooncall, notpayments-oncall. The configuration reference explains why: “The regex is anchored on both ends. To un-anchor the regex, use .*<regex>.* .” The patternpayments|billingmatches the whole valuepaymentsor the whole valuebilling, never a prefix.
The last three commands show the extras:
$ amtool config routes test --tree --config.file=routes.yml severity=page team=billing
Matching routes:
.
└── default-route
└── {severity="page"}
└── {team=~"payments|billing"} receiver: payments-oncall
payments-oncall
$ amtool config routes test --config.file=continue.yml severity=page
chat-mirror,oncall
$ amtool config routes test --config.file=routes.yml --verify.receivers=tickets severity=page
oncall
WARNING: Expected receivers did not match resolved receivers.
With --tree, amtool prints only the branch the alert took. The next command uses a second file, continue.yml, to show what continue: true changes:
# continue.yml: mirror every page into a chat channel and still page the on-call
route:
receiver: triage
routes:
- matchers: [severity="page"]
receiver: chat-mirror
continue: true
- matchers: [severity="page"]
receiver: oncall
receivers:
- name: triage
- name: chat-mirror
- name: oncall
The first route matches pages and sends them to chat-mirror, and because of continue: true the alert keeps going down the list and also matches the second route. The test prints both receivers, chat-mirror,oncall, in the order they appear in the file. Use this to copy pages to a chat channel while still paging a person. Finally, --verify.receivers=tickets says “I expect exactly this list”; when the real answer is oncall amtool prints a warning, and the exit status is 1 (I saw 1 when I ran it by hand). That is the hook the CI gate in step 10 builds on.
Common mistakes in this step. A broad route written before a narrower sibling matches first, and without continue: true the narrower route is never reached, so order matters (planted mistake 7 in step 10 shows it). A matcher value that contains spaces or punctuation needs double quotes, or Alertmanager falls back to its old parser and logs a warning like the one in step 2. And amtool config routes test tells you which receiver an alert reaches, but nothing about timing or about inhibition; later steps cover both.
Step 4: Know which labels your rules really produce
A routing tree can only match on the labels an alert actually carries, and they come from two places: the series that the alert’s query returned, and the labels: you write in the rule. One query function breaks the first source in a way that is easy to miss: absent(), the standard way to alert when a metric disappears. The Prometheus documentation shows its result: absent(nonexistent{job="myjob"}) gives {job="myjob"}. There is no series, so there are no labels to copy; absent() can only derive them from the equality matchers in the selector. If your routing depends on a team label that comes from your scrape configuration, an absent() alert will not have it.
You can prove this with promtool from the earlier post, without running Prometheus. Two files: a rule file with a normal alert and an absent() alert, and a unit test that feeds in a series that stops.
groups:
- name: checkout
rules:
- alert: CheckoutErrors
expr: rate(http_requests_failed_total{job="checkout"}[5m]) > 0.5
for: 2m
labels:
severity: page
service: checkout
- alert: CheckoutMetricsMissing
expr: absent(up{job="checkout"})
for: 2m
labels:
severity: page
service: checkout
rule_files:
- absent_rules.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'http_requests_failed_total{job="checkout", instance="a:9100", team="payments"}'
values: '0+60x20'
- series: 'up{job="checkout", instance="a:9100", team="payments"}'
values: '1x5 _x20'
alert_rule_test:
- eval_time: 8m
alertname: CheckoutErrors
exp_alerts:
- exp_labels:
severity: page
service: checkout
job: checkout
instance: "a:9100"
team: payments
- eval_time: 15m
alertname: CheckoutMetricsMissing
exp_alerts:
- exp_labels:
severity: page
service: checkout
job: checkout
The test supplies two series for the checkout job, both carrying team="payments". The error counter climbs one request per second for the whole test, which is above the 0.5 threshold. The up series has six samples, for minutes 0 to 5, and then nothing (the underscore means “no sample”). At 15 minutes absent() has been true for long enough to pass its two-minute for:. The test lists the labels each alert must have at the moment it is checked, and promtool compares the whole label set, so any label you leave out of the expectation makes the test fail. The script below runs it and then asks amtool where each alert would go, then fixes the rule and runs both again:
# s9_absent.py
"""s9_absent.py: which labels does an absent() alert carry, and where does Alertmanager send it?"""
import difflib
import os
import subprocess
from amlab import HERE, amtool, read
PROMTOOL = os.environ.get("PROMTOOL", "promtool")
def run(args):
p = subprocess.run([PROMTOOL, *args], capture_output=True, text=True, encoding="utf-8", cwd=HERE)
return (p.stdout + p.stderr).strip()
def route(labels):
out, _ = amtool("config", "routes", "test", "--config.file=routes.yml", *labels.split())
print(f"$ amtool config routes test --config.file=routes.yml {labels}\n{out}")
print("$ promtool test rules absent_test.yml")
print(run(["test", "rules", "absent_test.yml"]))
print()
route("alertname=CheckoutErrors severity=page service=checkout job=checkout instance=a:9100 team=payments")
route("alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout")
wrong = read("absent_test.yml") + " team: payments\n"
with open(f"{HERE}/absent_test_wrong.yml", "w", newline="\n") as f:
f.write(wrong)
print("$ promtool test rules absent_test_wrong.yml (the absent() alert is expected to carry team: payments)")
print("\n".join(line for line in run(["test", "rules", "absent_test_wrong.yml"]).splitlines() if line.strip())[:900])
print()
rules = read("absent_rules.yml")
fixed_rules = rules.replace(" service: checkout\n", " service: checkout\n team: payments\n")
test = read("absent_test.yml")
fixed_test = test.replace("absent_rules.yml", "absent_rules_fixed.yml") + " team: payments\n"
with open(f"{HERE}/absent_rules_fixed.yml", "w", newline="\n") as f:
f.write(fixed_rules)
with open(f"{HERE}/absent_test_fixed.yml", "w", newline="\n") as f:
f.write(fixed_test)
print()
print("--- diff absent_rules.yml absent_rules_fixed.yml")
print("".join(difflib.unified_diff(rules.splitlines(1), fixed_rules.splitlines(1), "absent_rules.yml", "absent_rules_fixed.yml", n=1)))
print("--- diff absent_test.yml absent_test_fixed.yml")
print("".join(difflib.unified_diff(test.splitlines(1), fixed_test.splitlines(1), "absent_test.yml", "absent_test_fixed.yml", n=1)))
print("$ promtool test rules absent_test_fixed.yml")
print(run(["test", "rules", "absent_test_fixed.yml"]))
print()
route("alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout team=payments")
Set the environment variable PROMTOOL to the path of promtool (or put it on your PATH) and run python s9_absent.py. I ran it with the 3.16.0 binary. Output:
$ promtool test rules absent_test.yml
Processing test file: absent_test.yml
SUCCESS
$ amtool config routes test --config.file=routes.yml alertname=CheckoutErrors severity=page service=checkout job=checkout instance=a:9100 team=payments
payments-oncall
$ amtool config routes test --config.file=routes.yml alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout
oncall
$ promtool test rules absent_test_wrong.yml (the absent() alert is expected to carry team: payments)
Processing test file: absent_test_wrong.yml
FAILED:
alertname: CheckoutMetricsMissing, time: 15m,
exp:[
0:
Labels:{alertname="CheckoutMetricsMissing", job="checkout", service="checkout", severity="page", team="payments"}
Annotations:{}
],
got:[
0:
Labels:{alertname="CheckoutMetricsMissing", job="checkout", service="checkout", severity="page"}
Annotations:{}
]
--- diff absent_rules.yml absent_rules_fixed.yml
--- absent_rules.yml
+++ absent_rules_fixed.yml
@@ -9,2 +9,3 @@
service: checkout
+ team: payments
- alert: CheckoutMetricsMissing
@@ -15 +16,2 @@
service: checkout
+ team: payments
--- diff absent_test.yml absent_test_fixed.yml
--- absent_test.yml
+++ absent_test_fixed.yml
@@ -1,3 +1,3 @@
rule_files:
- - absent_rules.yml
+ - absent_rules_fixed.yml
evaluation_interval: 1m
@@ -27 +27,2 @@
job: checkout
+ team: payments
$ promtool test rules absent_test_fixed.yml
Processing test file: absent_test_fixed.yml
SUCCESS
$ amtool config routes test --config.file=routes.yml alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout team=payments
payments-oncall
Read it in order. The first test passes: it says the absent() alert has severity, service and job and no team, and promtool agrees. The two routing lines show the consequence. The error alert, which has team=payments, goes to payments-oncall. The missing-metrics alert for the same service goes to plain oncall, and the payments team never hears about it. The third block is the proof that the missing label is real, not my assumption: when the test was told that the absent() alert should carry team: payments, promtool failed it and printed the expected and the actual label sets one above the other, and the actual set has no team.
The fix is to put every label that routing needs into the rule’s labels:, and not to rely on labels borrowed from series. The first diff shows the two added lines in the rule file, the second diff the added expectation, the fixed test passes, and the route test now prints payments-oncall for the absent() alert as well.
Rule of thumb: for every label your routing and inhibition rules use, ask where it comes from. If the answer is “the target”, an absent() alert will not have it. Write routing labels into the rule.
Step 5: Control how many notifications you get with group_by
When ten pods of one service all fail, you want one message that says “ten pods are failing”, not ten messages. That is grouping, and group_by decides it: alerts with identical values for the listed labels share a group, and each group sends one notification. The experiment below posts the same six alerts under five different group_by settings, each with its own Alertmanager running side by side. Three are HighLatency for the checkout service (pods p1 to p3), two are HighLatency for search (p1 and p2), and one is DiskFull for checkout.
# s4_group.py
"""s4_group.py: the same six alerts under five group_by settings."""
import threading
from amlab import Lab
TEMPLATE = """route:
receiver: sink
group_wait: 2s
{group_by}
receivers:
- name: sink
webhook_configs:
- url: http://127.0.0.1:9099/sink
"""
VARIANTS = [
" group_by: [alertname, service]",
" group_by: [alertname, service, pod]",
" group_by: [alertname, service, cluster]",
" group_by: ['...']",
" # no group_by line at all",
]
ALERTS = ([dict(alertname="HighLatency", service="checkout", pod=f"p{i}") for i in (1, 2, 3)]
+ [dict(alertname="HighLatency", service="search", pod=f"p{i}") for i in (1, 2)]
+ [dict(alertname="DiskFull", service="checkout", pod="p1")])
results = {}
def run(k, line):
lab = Lab(f"group{k}", TEMPLATE.format(group_by=line), k=k).start()
lab.post(*ALERTS)
lab.wait_until(6)
results[k] = [len(p["alerts"]) for _, _, p in lab.notifications()]
lab.stop()
threads = [threading.Thread(target=run, args=(k, line)) for k, line in enumerate(VARIANTS)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"{'group_by setting':<40}{'notifications':>14} alerts in each")
for k, line in enumerate(VARIANTS):
sizes = sorted(results[k], reverse=True)
print(f"{line.strip():<40}{len(sizes):>14} {', '.join(map(str, sizes))}")
Run python s4_group.py:
group_by setting notifications alerts in each
group_by: [alertname, service] 3 3, 2, 1
group_by: [alertname, service, pod] 6 1, 1, 1, 1, 1, 1
group_by: [alertname, service, cluster] 3 3, 2, 1
group_by: ['...'] 6 1, 1, 1, 1, 1, 1
# no group_by line at all 1 6
[alertname, service]gives three notifications of three, two and one alerts. This is usually the right choice: one message per kind of problem per service, listing every pod in it.- Adding
podgives six notifications, one per alert. You have grouped by something so specific that grouping does nothing. - Listing a label that no alert has, here
cluster, changes nothing: still three notifications. A label that is missing counts as an empty value, the same for every alert, so it does not split the groups and it does not raise an error. A typo in agroup_bylabel name therefore fails quietly in the direction of fewer, larger groups. ['...']is the special value that means group by every label, which the reference says “effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want”. The result is six notifications.- With no
group_byat all, every alert in the route lands in one group. One notification carries all six alerts. That can be right for a small system and wrong for a large one, because a single group also shares one set of timers.
Pick group_by labels that name what a human would act on: the kind of problem and the thing that owns it. Pod and instance names belong in the notification’s list of alerts, not in the group key.
Step 6: Understand the three timers by watching a clock
Grouping needs time to work: Alertmanager has to wait a moment so related alerts can arrive together, and it has to decide how often to speak again. Three settings control that, and the reference describes each of them.
group_wait: “How long to wait before sending the first notification for a new group of alerts.” It exists so that alerts from other rule groups, and any alert that should inhibit this one, can arrive first. The reference adds: “If an alert is resolved before group_wait has elapsed, no notification will be sent for that alert. This reduces noise of flapping alerts.”group_interval: how long to wait before sending news about a group that has already been notified. It is “a recurring timer that starts as soon as group_wait has elapsed”. At each tick Alertmanager sends a notification if alerts were added or resolved since the last one.repeat_interval: how long to wait before repeating a notification when nothing has changed. “Since the repeat_interval is checked after each group_interval, it should be a multiple of the group_interval. If it’s not, the repeat_interval is rounded up to the next multiple of the group_interval.”
The defaults are 30 seconds, 5 minutes and 4 hours, which makes experiments slow. The harness therefore shrinks every timer in routes.yml with the FAST table you saw in amlab.py: 30 seconds becomes 2, 10 seconds becomes 1, 5 minutes becomes 5 seconds, and the three repeat intervals become 60, 30 and 60 seconds. The experiment below sets its own timers directly: group_wait: 2s, group_interval: 5s, repeat_interval: 12s. Note that 12 is not a multiple of 5, on purpose. The script posts and resolves alerts at fixed times, then prints your actions and Alertmanager’s notifications on one timeline, with >> marking what the script did:
# s5_timers.py
"""s5_timers.py: watch group_wait, group_interval and repeat_interval on a clock."""
from datetime import datetime, timedelta, timezone
from amlab import Lab
from receiver import render
CONFIG = """route:
receiver: sink
group_by: [alertname, service]
group_wait: 2s
group_interval: 5s
repeat_interval: 12s
receivers:
- name: sink
webhook_configs:
- url: http://127.0.0.1:9099/sink
"""
A = dict(alertname="HighLatency", service="checkout", pod="p1")
B = dict(alertname="HighLatency", service="checkout", pod="p2")
FLAP = dict(alertname="HighLatency", service="search", pod="p1")
OLD = dict(alertname="HighLatency", service="billing", pod="p1")
lab = Lab("timers", CONFIG).start()
log = []
def at(t, text, action):
lab.wait_until(t)
action()
log.append((lab.now(), f"{lab.now():5.1f}s >> {text}"))
at(0, "post A (checkout p1)", lambda: lab.post(A))
at(3, "post B (checkout p2)", lambda: lab.post(B))
at(10, "post FLAP (search p1)", lambda: lab.post(FLAP))
at(11, "resolve FLAP, one second later", lambda: lab.resolve(FLAP))
at(14, "post OLD (billing p1) with startsAt 60 s in the past",
lambda: lab.post(OLD, started=datetime.now(timezone.utc) - timedelta(seconds=60)))
at(16, "resolve OLD", lambda: lab.resolve(OLD))
at(26, "resolve A", lambda: lab.resolve(A))
at(33, "resolve B", lambda: lab.resolve(B))
lab.wait_until(44)
rows = log + [(t, render(t, name, p)) for t, name, p in lab.notifications()]
for _, text in sorted(rows, key=lambda r: r[0]):
print(text)
lab.stop()
Run python s5_timers.py. It takes 44 seconds:
0.0s >> post A (checkout p1)
2.0s sink firing group={alertname=HighLatency service=checkout}
F alertname=HighLatency pod=p1 service=checkout
3.0s >> post B (checkout p2)
7.0s sink firing group={alertname=HighLatency service=checkout}
F alertname=HighLatency pod=p1 service=checkout
F alertname=HighLatency pod=p2 service=checkout
10.0s >> post FLAP (search p1)
11.0s >> resolve FLAP, one second later
14.0s >> post OLD (billing p1) with startsAt 60 s in the past
14.0s sink firing group={alertname=HighLatency service=billing}
F alertname=HighLatency pod=p1 service=billing
16.0s >> resolve OLD
19.0s sink resolved group={alertname=HighLatency service=billing}
R alertname=HighLatency pod=p1 service=billing
22.0s sink firing group={alertname=HighLatency service=checkout}
F alertname=HighLatency pod=p1 service=checkout
F alertname=HighLatency pod=p2 service=checkout
26.0s >> resolve A
27.0s sink firing group={alertname=HighLatency service=checkout}
R alertname=HighLatency pod=p1 service=checkout
F alertname=HighLatency pod=p2 service=checkout
33.0s >> resolve B
37.0s sink resolved group={alertname=HighLatency service=checkout}
R alertname=HighLatency pod=p2 service=checkout
Read the timeline row by row.
- 0.0 to 2.0 seconds. Alert A is posted and the first notification arrives two seconds later: that is
group_wait. - 3.0 to 7.0. Alert B joins the same group one second after the first notification went out. It does not get its own two second wait. It waits for the next tick of the group’s recurring timer, which is 7.0 (the first flush at 2.0 plus five seconds), and then goes out together with A.
- 10.0 and 11.0. A search alert fires and resolves one second later, before its
group_waitcould expire. No notification is ever sent for it, which is the flapping protection the reference describes. Look for the search group in the output: it is not there. - 14.0. This alert was posted with a start time 60 seconds in the past, and the notification came out at 14.0, with no two second wait. In the source of the dispatcher at the v0.35.0 tag, line 605 checks
alrt.StartsAt.Add(ag.opts.GroupWait).Before(now)when a group is created and logs “Alert is old enough for immediate flush, resetting timer to zero”. Sogroup_waitis counted from the alert’s own start time. An alert that has already been firing for longer thangroup_waitwhen its group is created is sent at once. The configuration reference does not mention this; I found it by measuring and confirmed it in the source. - 16.0 and 19.0. That alert is resolved at 16.0, and a
resolvednotification arrives at 19.0, the next tick after 14.0. Resolutions travel on the same ticks as new alerts. - 22.0. The checkout group is notified again although nothing changed. The last message went out at 7.0 and
repeat_intervalis 12 seconds, but Alertmanager only looks at ticks: at 12.0 it had been 5 seconds, at 17.0 it had been 10, at 22.0 it had been 15, so it repeated at 22.0. A twelve second setting behaved as fifteen. That is the rounding the reference describes. - 26.0 and 27.0. A is resolved at 26.0 and the next tick, 27.0, sends the group with
Rfor A andFfor B. The group’sstatusis stillfiringbecause B is. - 33.0 and 37.0. B is resolved just after the tick at 32.0, so the final
resolvedmessage waits for 37.0.
What this means for your own settings. Keep group_wait short on the routes that page (10 seconds in routes.yml), but not so short that an inhibiting alert from another rule group has no time to arrive; step 7 shows what happens when it is late. Choose repeat_interval as a multiple of group_interval, so the number you write is the number you get. And remember that a resolved message is sent on a tick, so “the page cleared” can reach the receiver up to one group_interval after the alert actually ended.
Step 7: Stop tickets from piling on a page with inhibition
When a service is down, you do not also want a ticket for its slow queries and a ticket for its high latency. The page already says everything. An inhibition rule expresses that. The one in routes.yml reads:
inhibit_rules:
- name: page-covers-ticket
source_matchers: [severity="page"]
target_matchers: [severity="ticket"]
equal: [service]
In words: while an alert matching the source matchers is firing, mute alerts matching the target matchers, but only when both alerts have the same value for every label in equal. So a page about service=checkout mutes tickets about checkout and leaves tickets about other services alone. The reference gives two rules that matter here. The first: “Semantically, a missing label and a label with an empty value are the same thing. Therefore, if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply.” The second concerns an alert that matches both sides: such an alert “cannot be inhibited by alerts for which the same is true (including itself)”, but the reference adds “we recommend to choose target and source matchers in a way that alerts never match both sides”, as this rule does.
The experiment runs the same eight alerts through two Alertmanagers, one with the equal line and one without. Three pages (P1, P0, P4) and five tickets (T0 to T4) are involved. P1 is a page for checkout and T1 a ticket for checkout, so T1 should be inhibited. T2 and T3 are tickets for search and billing and should get through. P0 and T0 are a page and a ticket that have no service label at all. T4 is a ticket for orders that arrives first, and P4, the page for orders, arrives 4.5 seconds later, after T4’s notification has already gone out. At 7.0 seconds the script reads Alertmanager’s own view of every alert, and at 8.5 it resolves P1.
# s6_inhibit.py
"""s6_inhibit.py: one inhibit rule, with and without its equal line."""
import threading
import time
from amlab import Lab, fast, print_notifications, read, state_line
P1 = dict(alertname="DiskFull", severity="page", service="checkout")
T1 = dict(alertname="SlowQueries", severity="ticket", service="checkout") # same service as P1
T2 = dict(alertname="SlowQueries", severity="ticket", service="search") # another service
T3 = dict(alertname="CertExpiry", severity="ticket", service="billing") # another service
P0 = dict(alertname="NodeDown", severity="page") # no service label
T0 = dict(alertname="NodeDiskWarn", severity="ticket") # no service label
T4 = dict(alertname="SlowQueries", severity="ticket", service="orders") # the ticket arrives first
P4 = dict(alertname="DiskFull", severity="page", service="orders") # the page arrives 4.5 s later
base = fast(read("routes.yml"))
configs = [("with equal: [service]", base),
("without the equal line", base.replace(" equal: [service]\n", ""))]
assert configs[0][1] != configs[1][1]
labs = [Lab(f"inhibit{k}", cfg, k=k).start() for k, (_, cfg) in enumerate(configs)]
states = {}
def run(lab):
lab.t0 = lab.sink.t0 = time.monotonic()
lab.post(P1, T1, T2, T3, P0, T0, T4)
lab.wait_until(4.5)
lab.post(P4)
lab.wait_until(7)
rows = sorted(lab.alerts(), key=lambda a: (a["labels"]["alertname"], a["labels"].get("service", "")))
states[lab.name] = [state_line(a) for a in rows]
lab.wait_until(8.5)
lab.resolve(P1)
lab.wait_until(16)
threads = [threading.Thread(target=run, args=(lab,)) for lab in labs]
for t in threads:
t.start()
for t in threads:
t.join()
for lab, (title, _) in zip(labs, configs):
print(f"=== {title}")
print("t=0.0s posted P1 T1 T2 T3 P0 T0 T4 together; t=4.5s posted P4; t=7.0s read /api/v2/alerts; t=8.5s resolved P1")
print_notifications(lab)
print("--- /api/v2/alerts at t=7.0s")
for line in states[lab.name]:
print(line)
print()
lab.stop()
Run python s6_inhibit.py. First the Alertmanager with the equal line:
t=0.0s posted P1 T1 T2 T3 P0 T0 T4 together; t=4.5s posted P4; t=7.0s read /api/v2/alerts; t=8.5s resolved P1
1.0s oncall firing group={alertname=NodeDown}
F alertname=NodeDown severity=page
1.0s oncall firing group={alertname=DiskFull service=checkout}
F alertname=DiskFull service=checkout severity=page
2.0s tickets firing group={alertname=SlowQueries service=orders}
F alertname=SlowQueries service=orders severity=ticket
2.0s tickets firing group={alertname=CertExpiry service=billing}
F alertname=CertExpiry service=billing severity=ticket
2.0s tickets firing group={alertname=SlowQueries service=search}
F alertname=SlowQueries service=search severity=ticket
5.5s oncall firing group={alertname=DiskFull service=orders}
F alertname=DiskFull service=orders severity=page
11.0s oncall resolved group={alertname=DiskFull service=checkout}
R alertname=DiskFull service=checkout severity=page
12.0s tickets firing group={alertname=SlowQueries service=checkout}
F alertname=SlowQueries service=checkout severity=ticket
--- /api/v2/alerts at t=7.0s
CertExpiry service=billing severity=ticket active
DiskFull service=checkout severity=page active
DiskFull service=orders severity=page active
NodeDiskWarn severity=ticket suppressed inhibited
NodeDown severity=page active
SlowQueries service=checkout severity=ticket suppressed inhibited
SlowQueries service=orders severity=ticket suppressed inhibited
SlowQueries service=search severity=ticket active
- T1 is held back and T2 and T3 get through. At 2.0 seconds the ticket queue receives only the search, billing and orders tickets. The checkout ticket T1 is missing. In the API view at 7.0 seconds it has state
suppressedwith reasoninhibited. - Two alerts with no
servicelabel inhibit each other. T0 (NodeDiskWarn) never reached the queue, because P0 (NodeDown) has noserviceeither, and a missing label equals an empty one. That is exactly the case the reference describes. Sometimes it is what you want. When it is not, give every alert aservicelabel in its rule, as step 4 recommends. - Order matters: T4 was sent before P4 arrived. The orders ticket went out at 2.0, because no page for orders existed yet. P4 arrived at 4.5 and from then on T4 shows as
suppressedandinhibitedin the API, but the message that already left is not recalled. This is why the reference tiesgroup_waitto inhibition: it is long enough “for one or more inhibiting alerts to arrive and mute any target alerts before the first notification”. If your page alert can fire a minute after the ticket alert, the ticket will be sent. - Inhibition ends when the source resolves. P1 was resolved at 8.5 seconds, and the held-back checkout ticket was delivered at 12.0, the next tick of its group. Until then it waited quietly; the page’s own
resolvedmessage arrived at 11.0.
Now the same alerts when the rule has no equal line:
t=0.0s posted P1 T1 T2 T3 P0 T0 T4 together; t=4.5s posted P4; t=7.0s read /api/v2/alerts; t=8.5s resolved P1
1.0s oncall firing group={alertname=NodeDown}
F alertname=NodeDown severity=page
1.0s oncall firing group={alertname=DiskFull service=checkout}
F alertname=DiskFull service=checkout severity=page
5.5s oncall firing group={alertname=DiskFull service=orders}
F alertname=DiskFull service=orders severity=page
11.0s oncall resolved group={alertname=DiskFull service=checkout}
R alertname=DiskFull service=checkout severity=page
--- /api/v2/alerts at t=7.0s
CertExpiry service=billing severity=ticket suppressed inhibited
DiskFull service=checkout severity=page active
DiskFull service=orders severity=page active
NodeDiskWarn severity=ticket suppressed inhibited
NodeDown severity=page active
SlowQueries service=checkout severity=ticket suppressed inhibited
SlowQueries service=orders severity=ticket suppressed inhibited
SlowQueries service=search severity=ticket suppressed inhibited
With no equal, the rule says “while any page is firing anywhere, mute every ticket”. All five tickets are suppressed, nothing at all reaches the queue, and no error or warning tells you so. In production this is one of the nastiest inhibition bugs, because the pager works and the tickets silently stop whenever any page is open. Always write the equal line, and use labels that every alert carries.
Step 8: Mute alerts on purpose with silences
A silence is the manual version of inhibition: a matcher, a start, an end, an author and a comment. You use it for planned work, so a maintenance window does not generate tickets. The reference says alerts are checked “whether they match all the equality or regular expression matchers of an active silence. If they do, no notifications will be sent out for that alert.” You can create silences in the web interface or with amtool. The script below uses amtool for every silence command, and it prints the commands as you would type them against the default port; the lab itself runs on another port. Silence IDs are random, so the script replaces them with <id>.
# s7_silence.py
"""s7_silence.py: silence an alert with amtool, test an unanchored regex, expire the silence."""
from amlab import Lab, amtool, fast, mask, print_notifications, read, state_line
lab = Lab("silence", fast(read("routes.yml"))).start()
URL = f"--alertmanager.url={lab.url}"
SHOWN = "--alertmanager.url=http://127.0.0.1:9093"
T1 = dict(alertname="SlowQueries", severity="ticket", service="checkout")
T2 = dict(alertname="SlowQueries", severity="ticket", service="search")
T3 = dict(alertname="CertExpiry", severity="ticket", service="billing")
T5 = dict(alertname="DiskWarn", severity="ticket", service="billing")
def run(*args):
out, rc = amtool(*args, URL)
print("$ amtool " + mask(" ".join(f"'{a}'" if " " in a or "~" in a else a for a in args)) + " " + SHOWN)
print(mask(out))
return out
def silence(*matchers, comment):
out = run("silence", "add", *matchers, f"--comment={comment}", "--author=lab", "--duration=10m")
return out.splitlines()[-1].strip()
def states(title):
print(f"--- /api/v2/alerts, {title}")
for a in sorted(lab.alerts(), key=lambda a: (a["labels"]["alertname"], a["labels"]["service"])):
print(state_line(a))
sid = silence("service=checkout", "severity=ticket", comment="planned database maintenance")
lab.post(T1, T2)
lab.wait_until(4)
states(f"t={lab.now():.1f}s")
run("silence", "query")
silence('service=~"bill"', comment="regex without a wildcard")
lab.post(T3)
lab.wait_until(8)
states(f"t={lab.now():.1f}s, after silencing service=~\"bill\" and posting CertExpiry for billing")
silence('service=~"bill.*"', comment="regex with a wildcard")
lab.post(T5)
lab.wait_until(12)
states(f"t={lab.now():.1f}s, after silencing service=~\"bill.*\" and posting DiskWarn for billing")
t_expire = lab.now()
run("silence", "expire", sid)
lab.wait_until(20)
print(f"--- webhook calls (silence for checkout expired at t={t_expire:.1f}s)")
print_notifications(lab)
lab.stop()
Run python s7_silence.py. It takes 20 seconds:
$ amtool silence add service=checkout severity=ticket '--comment=planned database maintenance' --author=lab --duration=10m --alertmanager.url=http://127.0.0.1:9093
<id>
--- /api/v2/alerts, t=4.0s
SlowQueries service=checkout severity=ticket suppressed silenced
SlowQueries service=search severity=ticket active
$ amtool silence query --alertmanager.url=http://127.0.0.1:9093
ID Matchers Ends At Created By Comment
<id> service="checkout" severity="ticket" 2026-11-15 20:18:35 UTC lab planned database maintenance
$ amtool silence add 'service=~"bill"' '--comment=regex without a wildcard' --author=lab --duration=10m --alertmanager.url=http://127.0.0.1:9093
<id>
--- /api/v2/alerts, t=8.0s, after silencing service=~"bill" and posting CertExpiry for billing
CertExpiry service=billing severity=ticket active
SlowQueries service=checkout severity=ticket suppressed silenced
SlowQueries service=search severity=ticket active
$ amtool silence add 'service=~"bill.*"' '--comment=regex with a wildcard' --author=lab --duration=10m --alertmanager.url=http://127.0.0.1:9093
<id>
--- /api/v2/alerts, t=12.0s, after silencing service=~"bill.*" and posting DiskWarn for billing
CertExpiry service=billing severity=ticket suppressed silenced
DiskWarn service=billing severity=ticket suppressed silenced
SlowQueries service=checkout severity=ticket suppressed silenced
SlowQueries service=search severity=ticket active
$ amtool silence expire <id> --alertmanager.url=http://127.0.0.1:9093
--- webhook calls (silence for checkout expired at t=12.0s)
2.0s tickets firing group={alertname=SlowQueries service=search}
F alertname=SlowQueries service=search severity=ticket
6.1s tickets firing group={alertname=CertExpiry service=billing}
F alertname=CertExpiry service=billing severity=ticket
17.0s tickets firing group={alertname=SlowQueries service=checkout}
F alertname=SlowQueries service=checkout severity=ticket
- The exact silence works. After
silence add service=checkout severity=ticket, the checkout ticket issuppressedwith reasonsilencedand the search ticket, which the silence does not match, is delivered at 2.0 seconds.amtool silence querylists the silence with its matchers, end time, author and comment. - A regular expression without a wildcard silences nothing.
service=~"bill"was meant to catch the billing service, but the regular expression is anchored on both ends, so it only matches the valuebill. The billing ticketCertExpirystayedactiveand its notification went out at 6.1 seconds. Onlyservice=~"bill.*"matched, and it silenced both billing alerts, including the one that had already been notified. A silence stops future notifications; it does not recall one that was sent. - Expiring a silence does not deliver at once. The silence on checkout was expired at 12.0 seconds and the held-back ticket arrived at 17.0: the next tick of its group, five seconds later.
Good habits that follow from this: match on the narrowest labels that describe the work (service and severity here, never alertname alone), always give an author and a comment so the next person knows why the alert is quiet, and set a duration so it expires by itself. Check the result in the alert list; a silence that matches nothing looks exactly like one that works until the alert fires.
Step 9: Keep tickets quiet outside working hours with time intervals
A route can also be muted by the clock. You define named time_intervals and attach them to routes with mute_time_intervals (do not notify during these times) or active_time_intervals (notify only during these times). The reference says of muting: “When a route is muted it will not send any notifications, but otherwise acts normally (including ending the route-matching process if the continue option is not set.)” It also contains a sentence worth testing: “mute_time_intervals is not inherited from the parent route. If omitted, the route has no mute time intervals, even if the parent specifies some.” That is unlike the timers and group_by, which children do inherit.
The experiment mutes the whole ticket route all day (an interval from 00:00 to 24:00, so the result does not depend on the time you run it) and adds a child route for team=db that does not mention muting. Three alerts go in: a page, a ticket for search, and a ticket for the db team.
# s8_mute.py
"""s8_mute.py: mute_time_intervals on a route, and the fact that a child route does not inherit them."""
import os
import sys
import urllib.request
from amlab import HERE, Lab, amtool, print_notifications, state_line
sys.stdout.reconfigure(encoding="utf-8")
CONFIG = """route:
receiver: triage
group_by: [alertname, service]
group_wait: 2s
group_interval: 5s
routes:
- matchers: [severity="page"]
receiver: oncall
- matchers: [severity="ticket"]
receiver: tickets
mute_time_intervals: [all-day]
routes:
- matchers: [team="db"]
receiver: db-tickets
time_intervals:
- name: all-day
time_intervals:
- times:
- start_time: '00:00'
end_time: '24:00'
receivers:
- name: triage
webhook_configs:
- url: http://127.0.0.1:9099/triage
- name: oncall
webhook_configs:
- url: http://127.0.0.1:9099/oncall
- name: tickets
webhook_configs:
- url: http://127.0.0.1:9099/tickets
- name: db-tickets
webhook_configs:
- url: http://127.0.0.1:9099/db-tickets
"""
PAGE = dict(alertname="DiskFull", severity="page", service="checkout")
TICKET = dict(alertname="SlowQueries", severity="ticket", service="search")
DB_TICKET = dict(alertname="SlowQueries", severity="ticket", service="orders", team="db")
lab = Lab("mute", CONFIG).start()
lab.post(PAGE, TICKET, DB_TICKET)
lab.wait_until(7)
print("--- webhook calls")
print_notifications(lab)
print("--- /api/v2/alerts at t=7.0s")
for a in sorted(lab.alerts(), key=lambda a: (a["labels"]["alertname"], a["labels"]["service"])):
print(state_line(a))
lab.stop()
print()
out, rc = amtool("check-config", "hours.yml")
print("$ amtool check-config hours.yml")
print(out)
ZIP = f"{HERE}/zoneinfo.zip"
if not os.path.exists(ZIP):
urllib.request.urlretrieve("https://raw.githubusercontent.com/golang/go/master/lib/time/zoneinfo.zip", ZIP)
out, rc = amtool("check-config", "hours.yml", env={"ZONEINFO": ZIP})
print()
print("$ ZONEINFO=zoneinfo.zip amtool check-config hours.yml")
print(out)
The second half of the script checks a realistic working-hours file, hours.yml, which uses active_time_intervals and a time zone:
# hours.yml: tickets reach the queue only during working hours
route:
receiver: triage
routes:
- matchers: [severity="ticket"]
receiver: tickets
active_time_intervals: [working-hours]
time_intervals:
- name: working-hours
time_intervals:
- weekdays: ['monday:friday']
times:
- start_time: '09:00'
end_time: '18:00'
location: 'America/New_York'
receivers:
- name: triage
- name: tickets
Run python s8_mute.py:
--- webhook calls
2.0s db-tickets firing group={alertname=SlowQueries service=orders}
F alertname=SlowQueries service=orders severity=ticket team=db
2.0s oncall firing group={alertname=DiskFull service=checkout}
F alertname=DiskFull service=checkout severity=page
--- /api/v2/alerts at t=7.0s
DiskFull service=checkout severity=page active
SlowQueries service=orders severity=ticket team=db active
SlowQueries service=search severity=ticket active
$ amtool check-config hours.yml
Checking 'hours.yml' FAILED: unknown time zone America/New_York (on Windows platforms, you may have to pass the time zone database using the ZONEINFO environment variable, see https://pkg.go.dev/time#LoadLocation for details)
amtool.exe: error: failed to validate 1 file(s)
$ ZONEINFO=zoneinfo.zip amtool check-config hours.yml
Checking 'hours.yml' SUCCESS
Found:
- global config
- route
- 0 inhibit rules
- 2 receivers
- 0 templates
- The muted route sends nothing, and a child route is not muted with it. The page was delivered, the db team’s ticket was delivered through the child route
db-tickets, and the search ticket matched the mutedticketsroute and was not sent. The documentation’s sentence about inheritance held. - The API still calls the muted alert
active. In the alert list the search ticket looks like any other firing alert. A time mute stops the notification but, in this run, did not show up assuppressed, so an alert that is “active” in the list is not proof that anyone was told. Check the receiver, not the list. - Windows needs a time zone database for
location. The first check ofhours.ymlfailed with “unknown time zone America/New_York (on Windows platforms, you may have to pass the time zone database using the ZONEINFO environment variable”. Go’s documentation forLoadLocationsays it looks at “the directory or uncompressed zip file named by the ZONEINFO environment variable”. The script downloadedzoneinfo.zipfrom the Go repository (a stored, uncompressed zip of about 400 KB), pointedZONEINFOat it, and the same check then printedSUCCESS. On Linux and macOS Go reads the system time zone database, so no extra step is normally needed. When you run the real Alertmanager on Windows, set the same variable in its environment.
I did not test what happens to an alert that fires while its route is muted once the muting interval ends, so check that with your own clock before you rely on “tickets arrive at 09:00”.
Step 10: Put the routing tree under a CI gate and test the gate itself
Everything above can be checked by hand, and every configuration change deserves the same checks without anyone remembering to do them. The gate has two parts. amtool check-config catches files that Alertmanager would refuse to load. Then a table of cases, each a set of labels and the receiver you expect, goes through amtool config routes test. Any difference fails the build.
# gate.py
"""gate.py: fail unless the routing tree sends every test case to the receiver we expect.
usage: python gate.py alertmanager.yml
"""
import sys
from amlab import amtool
CASES = [
("payments page", "alertname=PaymentsErrors severity=page service=checkout team=payments", "payments-oncall"),
("billing page", "alertname=PaymentsErrors severity=page service=checkout team=billing", "payments-oncall"),
("other team page", "alertname=DiskFull severity=page service=search team=search", "oncall"),
("page without team", "alertname=NodeDown severity=page", "oncall"),
("ticket", "alertname=SlowQueries severity=ticket service=search", "tickets"),
("severity typo falls to triage", "alertname=DiskFull severity=critical service=search", "triage"),
("no severity falls to triage", "alertname=NodeDown", "triage"),
("team prefix is not a match", "alertname=PaymentsErrors severity=page team=payments-eu", "oncall"),
]
def check(config, cwd="."):
out, rc = amtool("check-config", config, cwd=cwd)
if rc != 0:
line = next((x for x in out.splitlines() if "FAILED" in x or "rror" in x), out.splitlines()[0])
return ["check-config failed: " + line.strip()]
problems = []
for name, labels, want in CASES:
out, rc = amtool("config", "routes", "test", f"--config.file={config}", *labels.split(), cwd=cwd)
if out.strip() != want:
problems.append(f"{name}: expected {want}, got {out.strip()}")
return problems
if __name__ == "__main__":
problems = check(sys.argv[1])
print("\n".join(problems) or f"all {len(CASES)} routing cases pass")
sys.exit(1 if problems else 0)
Run python gate.py routes.yml. The expected output is one line:
all 8 routing cases pass
A test that cannot fail is worthless, so the next script plants fourteen mistakes in routes.yml, one at a time, and runs the gate against each. Eleven are things people really do: a typo in a matcher value, a regular expression written as a prefix, a catch-all route placed first, a misspelled receiver. Three are changes that the gate has no way to see, and I included them to measure that blind spot honestly.
# mutants.py
"""mutants.py: plant mistakes in routes.yml one at a time and see which ones gate.py catches."""
import os
from amlab import HERE, read
from gate import check
BASE = read("routes.yml")
PAGE_ROUTE = ' - matchers: [severity="page"]\n'
MUTANTS = [
("page matcher typo", '- matchers: [severity="page"]', '- matchers: [severity="pager"]'),
("ticket matcher typo", '- matchers: [severity="ticket"]', '- matchers: [severity="tickets"]'),
("regex loses billing", 'team=~"payments|billing"', 'team=~"payments"'),
("regex written as a prefix", 'team=~"payments|billing"', 'team=~"pay|billing"'),
("pages go to the ticket queue", "receiver: oncall", "receiver: tickets"),
("payments child sends to oncall", "receiver: payments-oncall", "receiver: oncall"),
("catch-all route placed first", " routes:\n" + PAGE_ROUTE, " routes:\n - receiver: tickets\n" + PAGE_ROUTE),
("receiver name misspelled", "receiver: oncall", "receiver: oncal"),
("duration without a unit", "group_wait: 10s", "group_wait: 10"),
("root receiver is the ticket queue", "route:\n receiver: triage", "route:\n receiver: tickets"),
("page route loses its child", ' routes:\n - matchers: [team=~"payments|billing"]\n receiver: payments-oncall\n', ""),
("page repeats every second", "repeat_interval: 1h", "repeat_interval: 1s"),
("inhibit rule loses its equal line", " equal: [service]\n", ""),
("grouping switched off", "group_by: [alertname, service]", "group_by: ['...']"),
]
os.makedirs(f"{HERE}/work/mutants", exist_ok=True)
caught = 0
print(f"{'#':>2} {'planted mistake':<36}{'verdict':<10}what the gate said")
for n, (name, old, new) in enumerate(MUTANTS, 1):
assert old in BASE, name
path = f"{HERE}/work/mutants/m{n:02d}.yml"
with open(path, "w", newline="\n", encoding="utf-8") as f:
f.write(BASE.replace(old, new, 1))
problems = check(f"m{n:02d}.yml", cwd=f"{HERE}/work/mutants")
caught += bool(problems)
said = problems[0] if len(problems) == 1 else (f"{len(problems)} cases fail, first: {problems[0]}" if problems else "")
print(f"{n:>2} {name:<36}{'caught' if problems else 'SURVIVED':<10}{said}")
print(f"\nthe unmodified file: {check('routes.yml', cwd=HERE) or 'passes'}")
print(f"caught {caught} of {len(MUTANTS)}")
Run python mutants.py:
# planted mistake verdict what the gate said
1 page matcher typo caught 5 cases fail, first: payments page: expected payments-oncall, got triage
2 ticket matcher typo caught ticket: expected tickets, got triage
3 regex loses billing caught billing page: expected payments-oncall, got oncall
4 regex written as a prefix caught payments page: expected payments-oncall, got oncall
5 pages go to the ticket queue caught 3 cases fail, first: other team page: expected oncall, got tickets
6 payments child sends to oncall caught 2 cases fail, first: payments page: expected payments-oncall, got oncall
7 catch-all route placed first caught 7 cases fail, first: payments page: expected payments-oncall, got tickets
8 receiver name misspelled caught check-config failed: Checking 'm08.yml' FAILED: undefined receiver "oncal" used in route
9 duration without a unit caught check-config failed: Checking 'm09.yml' FAILED: not a valid duration string: "10"
10 root receiver is the ticket queue caught 2 cases fail, first: severity typo falls to triage: expected triage, got tickets
11 page route loses its child caught 2 cases fail, first: payments page: expected payments-oncall, got oncall
12 page repeats every second SURVIVED
13 inhibit rule loses its equal line SURVIVED
14 grouping switched off SURVIVED
the unmodified file: passes
caught 11 of 14
- Nine mistakes were caught by the routing cases. A misspelled
severityvalue fails five cases, a catch-all placed first fails seven (it swallows every alert before the real routes see it), and a regular expression written as the prefixpayfails the payments case. - Two were caught by
check-configalone. A receiver name with a letter missing is rejected with “undefined receiver”, and agroup_waitwithout a unit with “not a valid duration string”. Those files cannot even load, so the routing cases never run. - Three survived. A page route that repeats every second, an inhibit rule without its
equalline, and grouping switched off all pass both checks, becauseamtool config routes testonly reports which receiver an alert reaches. It says nothing about timers, grouping or inhibition. The inhibit change is the dangerous one, since step 7 showed it silencing every ticket.
What would catch the three survivors? Experiments like steps 5 to 7: start a throwaway Alertmanager with the candidate configuration, post a fixed set of alerts, and assert on the notifications that reach the sink. That takes about fifteen to twenty seconds per scenario (the inhibition experiment above ran for 16 seconds), which is acceptable in CI for a file this important. A cheaper partial check is to compare the timers and the inhibit_rules block against a reviewed copy and require a human approval when they change.
To use the gate in a pipeline, keep gate.py and its case table in the repository next to the configuration, and run bin/amtool check-config alertmanager.yml and python gate.py alertmanager.yml in a job that must pass before a merge. The script exits with status 1 and prints one line per failing case.
Common mistakes, in one place
- A label spelled differently from the route’s matcher (
criticalagainstpage,Pageagainstpage) sends the alert to the catch-all route, quietly. Make the catch-all receiver one that people read, and add cases to your gate for each label value you rely on. - A regular expression matcher is anchored on both ends.
team=~"pay"does not matchpayments, in routes or in silences. absent()alerts do not carry the labels of the missing series. Write the labels your routing needs into the rule.- A
group_bylabel that no alert has changes nothing and raises no error;['...']turns grouping off. - A
repeat_intervalthat is not a multiple ofgroup_intervalis rounded up. - An inhibit rule with no
equalline mutes every target alert while any source alert is firing. Alerts missing everyequallabel inhibit each other. - An inhibiting alert that arrives after
group_waitis too late to stop the first notification, and nothing recalls it. - A silence does not recall a notification that already went out; it only stops later ones.
- A muted route still shows its alerts as
activein the alert list, and a child route does not inherit its parent’s mute intervals.
Confirm that everything works end to end
Run the steps in order from one folder that holds bin, the Python files and the YAML files, and compare what you see with this post. python s2_first.py should print one firing notification after about 2 seconds. python s3_routes.py should print SUCCESS for the file, the tree, and the receivers in the cases above. python s4_group.py should print the 3, 6, 3, 6 and 1 notification counts. python s5_timers.py should show notifications at about 2, 7, 14, 19, 22, 27 and 37 seconds; the clock readings can differ by a tenth of a second on your machine. python s6_inhibit.py and python s7_silence.py should show the same suppressed and delivered alerts. python gate.py routes.yml should print “all 8 routing cases pass”, and python mutants.py should end with “caught 11 of 14”. The IDs and wall-clock timestamps in the JSON will differ; the receivers, group labels and counts should not.
Where to go next
- Connect a real Prometheus. Prometheus finds Alertmanager through the
alerting:section of its configuration. The overview page warns: “It’s important not to load balance traffic between Prometheus and its Alertmanagers, but instead, point Prometheus to a list of all Alertmanagers.” If you run more than one instance, turn the clustering that this lab disabled back on; the README lists the--cluster.*flags. - Replace the sink with a real receiver (PagerDuty, Slack, Opsgenie and others have their own settings), keep the credentials out of the repository, and read the post on verifying webhook signatures if your own service receives these webhooks.
- Add a heartbeat. Create an alert that always fires and route it to an external service that complains when it stops arriving. It is the only way to learn that the pipeline itself has died.
- Write notification templates so each message states the service, the runbook link and what to do first.
- Test the alert rules too. The earlier SLO post shows how to unit-test rules with promtool, so that both halves, “does the alert fire” and “does it reach the right person”, are covered by tests.
Sources and further reading
- Alertmanager configuration reference: routes, timers, inhibition, time intervals, matchers and the webhook payload.
- Alertmanager overview: grouping, inhibition, silences and high availability.
- Alertmanager v0.35.0 release and the README for the
amtoolexamples and the cluster flags. - dispatch.go at v0.35.0, the immediate-flush check for old alerts.
- Prometheus query functions: absent().
- Go time.LoadLocation, which explains
ZONEINFO, and the zoneinfo.zip in the Go repository.







No Comment! Be the first one.