TRENDING
A black manual telephone switchboard with rows of orange jacks and red patch cords plugged into the lower row, above a rotary dial and a handset
November 16, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
A brass nail with a threaded screw section lying on graph paper, a fastener that is both nail and screw
November 16, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
An amber warning beacon on a white surface, a flashing-light stand-in for the FBI’s FLASH alert
November 16, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query
Two eggs and a colour-changing egg timer in a saucepan of water on an induction hob
November 16, 2026
How to Set gRPC Deadlines and Retries in Python, Pass Them Down a Call Chain, and Stop Retries From Charging a Card Twice
A whole lemon and a halved lemon side by side on a black background
November 16, 2026
IBM’s LLMON Turns Prompt-Injection Defense Into a Token Layout, but the “Nearly Zero” Chart Is One Benchmark, the Paper’s Own Red Team Lets 15% to 20% of Indirect Attacks Through, and the Shipped Template Lets a Document Type the Delimiters
16 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
A manual paper cutter with a wooden grid board, a clear plastic guard and a long cutting arm, a picture of text being trimmed to fit
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
A brown llama and her gray cria walking across a salt flat beside a red lake in Bolivia
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
A stack of chocolate chip cookies on a white plate beside a cup of tea
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 16, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 302 Posts
News 307 Posts
Learning Hub 274 Posts
Home/Learning Hub/How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
Learning Hub

How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does

Run a real Alertmanager 0.35.0 on your own machine, build a routing tree that sends pages and tickets to the right receivers, measure grouping, timers, inhibition, silences and muted time intervals,...

November 15, 2026 40 Min Read
18

Prometheus decides when an alert fires. It does not decide who hears about it. That second job belongs to Alertmanager, and it is where many monitoring setups quietly go wrong. A page that should have woken the on-call engineer lands in a ticket queue nobody reads until Monday. A harmless warning wakes someone at 3 a.m. because a label was spelled differently. One broken database produces forty separate messages, and the one that matters is buried among them.

Table Of Content

  • What you will build
  • The words you need first
  • Prerequisites
  • Step 1: Download Alertmanager and check the download
  • Step 2: Send a first alert and read what arrives
  • Step 3: Build the routing tree and test it without sending anything
  • Step 4: Know which labels your rules really produce
  • Step 5: Control how many notifications you get with group_by
  • Step 6: Understand the three timers by watching a clock
  • Step 7: Stop tickets from piling on a page with inhibition
  • Step 8: Mute alerts on purpose with silences
  • Step 9: Keep tickets quiet outside working hours with time intervals
  • Step 10: Put the routing tree under a CI gate and test the gate itself
  • Common mistakes, in one place
  • Confirm that everything works end to end
  • Where to go next
  • Sources and further reading

In this tutorial you will run a real Alertmanager on your own machine, send it alerts, and watch what it does with them. You will build a routing tree that sends pages to an on-call receiver, sends payment problems to the payments team, and puts tickets in a queue. You will learn what grouping, the three timers, inhibition, silences and time-based muting actually do, with measurements instead of guesses. Then you will write a small check that runs in CI, plant fourteen mistakes in the configuration, and see which ones the check catches and which three it cannot see.

Nothing here needs a Prometheus server, a Kubernetes cluster, a cloud account or a paid service. A short Python program plays the part of PagerDuty, Slack or a ticket system. The alerts in this post carry the same severity labels, page and ticket, as the burn-rate alerts in the earlier post How to Alert on Your SLO Without Paging for Noise, so you can treat this one as the next stage of the same pipeline: that post makes the alerts fire at the right time, and this one gets them to the right place.

What you will build

  • a routing tree in routes.yml with four receivers: oncall, payments-oncall, tickets and a catch-all called triage;
  • a webhook receiver and a small harness that start a real Alertmanager 0.35.0, post alerts to it and print what arrives, with a timestamp;
  • one experiment each for grouping, the three timers, inhibition, silences and muted time intervals;
  • a CI gate, gate.py, that fails when an alert would go to the wrong receiver, plus a mutation check that plants fourteen mistakes to measure what the gate can and cannot see.

The words you need first

Alertmanager has its own small vocabulary. Read these once and the rest of the post will make sense.

  • Alert. A record that Prometheus (or any program) sends to Alertmanager. It has labels, which are name and value pairs such as severity=page, optional annotations, which are free text such as a summary, and a start time. The set of labels is the alert’s identity: two alerts with the same labels are the same alert.
  • Receiver. A named destination, such as a pager, a chat channel or a webhook URL. Alertmanager ships integrations for many services; this post uses the generic webhook one.
  • Routing tree. A tree of routes. Each route can have matchers (conditions on labels such as severity="page"), a receiver, timers, and child routes. Every alert enters at the root and is handed to the receiver of the deepest route that matches it.
  • Group. Alerts that share a route and the same values for the labels listed in group_by. Alertmanager sends one notification per group, not one per alert.
  • Inhibition. A rule that mutes some alerts while another alert is firing, for example “do not send tickets about a service while a page about the same service is open”.
  • Silence. A temporary mute that you create by hand, for example during planned maintenance.
  • Notification. What a receiver gets: one message for one group, listing every firing and recently resolved alert in it.

The configuration reference puts the first idea this way: “Grouping categorizes alerts of similar nature into a single notification.” It describes inhibition as “suppressing notifications for certain alerts if certain other alerts are already firing” and silences as “a straightforward way to simply mute alerts for a given time”. Those three sentences are the whole feature set you will be testing.

Prerequisites

  • Operating system: tested on Windows 11 with Git Bash. The same releases have Linux and macOS archives. I did not test those, and the only change the lab needs is dropping .exe from two names in amlab.py.
  • Alertmanager and amtool: version 0.35.0, released on November 9, 2026. The download is one archive of about 44 MB that holds both programs.
  • Python: 3.13 (tested with 3.13.14). The lab uses only the standard library.
  • promtool 3.16.0: only for step 4. The earlier SLO post shows how to install it, and you can skip step 4 without losing anything else.
  • Prior knowledge: you can read YAML and you know that a metric has labels. You do not need to have run Alertmanager before.
  • Time: about 45 minutes, most of it reading. The longest experiment runs 44 seconds.

Step 1: Download Alertmanager and check the download

Alertmanager and its command line tool, amtool, are single programs with no installer. You will download the archive, compare its checksum with the one the project publishes, and copy the two programs into a bin folder. Run this in Git Bash (or any POSIX shell):

mkdir am-lab && cd am-lab
curl -L -o am.zip https://github.com/prometheus/alertmanager/releases/download/v0.35.0/alertmanager-0.35.0.windows-amd64.zip
curl -L -o sha256sums.txt https://github.com/prometheus/alertmanager/releases/download/v0.35.0/sha256sums.txt
grep "windows-amd64.zip" sha256sums.txt
sha256sum am.zip
python -m zipfile -e am.zip unpacked
mkdir bin
cp unpacked/alertmanager-0.35.0.windows-amd64/alertmanager.exe unpacked/alertmanager-0.35.0.windows-amd64/amtool.exe bin/
bin/alertmanager.exe --version 2>&1 | head -1
bin/amtool.exe --version 2>&1 | head -1

The two checksum lines must be identical. The version lines should look like this:

cdf8cd664889bf3f27771881e10814f3d5916310aba90717b4424deb358b0fa3  alertmanager-0.35.0.windows-amd64.zip
cdf8cd664889bf3f27771881e10814f3d5916310aba90717b4424deb358b0fa3 *am.zip
alertmanager, version 0.35.0 (branch: HEAD, revision: 8cf67a8e75004bad6c3c6bacc829945cf8a3ee65)
amtool, version 0.35.0 (branch: HEAD, revision: 8cf67a8e75004bad6c3c6bacc829945cf8a3ee65)

If the checksums differ, delete the file and download it again; do not run a program whose checksum does not match. For Linux or macOS pick the archive for your platform from the v0.35.0 release page and compare against the same sha256sums.txt.

Step 2: Send a first alert and read what arrives

Before the routing tree, you need to see the whole round trip once: an alert goes in, a notification comes out. Alertmanager’s generic webhook receiver POSTs a JSON document to a URL. So the lab needs something that listens on a URL, remembers what arrives and prints it. That is receiver.py, about 50 lines of standard library Python:

# receiver.py
"""receiver.py: a stand-in for PagerDuty, Slack or a ticket system.

It accepts Alertmanager webhook POSTs, remembers them, and prints one block per
notification. Run it on its own with:  python receiver.py 9099
"""
import json
import sys
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


def short(labels):
    return " ".join(f"{k}={v}" for k, v in sorted(labels.items()))


def render(t, name, payload):
    head = f"{t:5.1f}s  {name:<16} {payload['status']:<8} group={{{short(payload['groupLabels'])}}}"
    lines = [head]
    for a in payload["alerts"]:
        lines.append(f"          {a['status'][:1].upper()}  {short(a['labels'])}")
    return "\n".join(lines)


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        size = int(self.headers.get("Content-Length", 0))
        payload = json.loads(self.rfile.read(size))
        name = self.path.strip("/")
        self.server.log.append((time.monotonic(), name, payload))
        self.send_response(200)
        self.end_headers()
        if self.server.echo:
            print(render(time.monotonic() - self.server.t0, name, payload), flush=True)

    def log_message(self, *args):
        pass


class Sink(ThreadingHTTPServer):
    daemon_threads = True

    def __init__(self, port, echo=False):
        super().__init__(("127.0.0.1", port), Handler)
        self.log = []
        self.echo = echo
        self.t0 = time.monotonic()


if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 9099
    print(f"listening on 127.0.0.1:{port}", flush=True)
    Sink(port, echo=True).serve_forever()

The Sink class is a normal HTTP server that stores every POST body together with a clock reading. render turns one notification into a readable block: the time since the start, the receiver’s name, whether the group is firing or resolved, the group’s labels, and then one line per alert with F for firing or R for resolved.

The second helper, amlab.py, is the harness every later step uses. It writes a configuration into its own folder, starts a real alertmanager.exe on its own port, starts a Sink for the webhooks, and offers four calls: post to send alerts, resolve to end them, alerts to read Alertmanager’s own view, and notifications to read what the sink received. It also holds the table that shrinks the timers, which you will meet in step 6. Save it next to receiver.py:

# amlab.py
"""amlab.py: start a real Alertmanager next to a webhook sink, then talk to both.

Windows, Python 3.13, standard library only. Alertmanager 0.35.0 lives in bin/.
"""
import json
import os
import re
import shutil
import subprocess
import threading
import time
import urllib.request
from datetime import datetime, timedelta, timezone

from receiver import Sink, render

HERE = os.path.dirname(os.path.abspath(__file__)).replace("\\", "/")
AM = f"{HERE}/bin/alertmanager.exe"
AMTOOL = f"{HERE}/bin/amtool.exe"
N = int(os.environ.get("LAB_N", "0"))  # lets several scripts run side by side on different ports

# The lab shrinks every timer so that minutes become seconds.
FAST = [
    ("group_wait: 30s", "group_wait: 2s"),
    ("group_wait: 10s", "group_wait: 1s"),
    ("group_interval: 5m", "group_interval: 5s"),
    ("repeat_interval: 4h", "repeat_interval: 60s"),
    ("repeat_interval: 1h", "repeat_interval: 30s"),
    ("repeat_interval: 24h", "repeat_interval: 60s"),
]


def read(name):
    with open(f"{HERE}/{name}", encoding="utf-8") as f:
        return f.read()


def fast(text):
    for old, new in FAST:
        assert old in text, old
        text = text.replace(old, new)
    return text


def amtool(*args, cwd=HERE, env=None):
    p = subprocess.run([AMTOOL, *args], capture_output=True, text=True, encoding="utf-8", cwd=cwd,
                       env=dict(os.environ, **env) if env else None)
    return (p.stdout + p.stderr).strip(), p.returncode


def mask(text):
    return re.sub(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", "<id>", text)


def stamp(dt):
    return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")


def state_line(a):
    s = a["status"]
    why = [name for name, key in (("inhibited", "inhibitedBy"), ("silenced", "silencedBy"), ("muted", "mutedBy")) if s.get(key)]
    labels = " ".join(f"{k}={v}" for k, v in sorted(a["labels"].items()) if k != "alertname")
    return f"{a['labels']['alertname']:<13}{labels:<42}{s['state']:<11}{'+'.join(why)}"


class Lab:
    def __init__(self, name, config, k=0, ports=None, echo=False):
        self.name = name
        self.am_port, self.rx_port = ports or (9100 + 10 * N + k, 9200 + 10 * N + k)
        self.dir = f"{HERE}/work/{name}"
        shutil.rmtree(self.dir, ignore_errors=True)
        os.makedirs(self.dir)
        self.cfg = f"{self.dir}/alertmanager.yml"
        with open(self.cfg, "w", newline="\n", encoding="utf-8") as f:
            f.write(config.replace("9099", str(self.rx_port)))
        self.url = f"http://127.0.0.1:{self.am_port}"
        self.sink = Sink(self.rx_port, echo=echo)
        threading.Thread(target=self.sink.serve_forever, daemon=True).start()
        self.proc = None

    def start(self):
        self.out = open(f"{self.dir}/alertmanager.log", "w")
        self.proc = subprocess.Popen(
            [AM, f"--config.file={self.cfg}", f"--storage.path={self.dir}/data",
             f"--web.listen-address=127.0.0.1:{self.am_port}", f"--web.external-url=http://localhost:{self.am_port}",
             "--cluster.listen-address="],
            stdout=self.out, stderr=subprocess.STDOUT)
        for _ in range(150):
            try:
                urllib.request.urlopen(self.url + "/-/ready", timeout=1)
                break
            except Exception:
                time.sleep(0.1)
        self.t0 = time.monotonic()
        self.sink.t0 = self.t0
        return self

    def now(self):
        return time.monotonic() - self.t0

    def wait_until(self, t):
        time.sleep(max(0.0, t - self.now()))

    def post(self, *label_sets, started=None, ended=None):
        now = datetime.now(timezone.utc)
        body = []
        for labels in label_sets:
            alert = {"labels": labels, "annotations": {"summary": labels["alertname"]},
                     "startsAt": stamp(started or now)}
            if ended:
                alert["endsAt"] = stamp(ended)
            body.append(alert)
        req = urllib.request.Request(self.url + "/api/v2/alerts", data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
        urllib.request.urlopen(req, timeout=5).read()

    def resolve(self, *label_sets):
        now = datetime.now(timezone.utc)
        self.post(*label_sets, started=now - timedelta(seconds=1), ended=now)

    def alerts(self):
        return json.load(urllib.request.urlopen(self.url + "/api/v2/alerts", timeout=5))

    def notifications(self):
        return [(t - self.t0, name, p) for t, name, p in self.sink.log]

    def stop(self):
        if self.proc:
            self.proc.terminate()
            self.proc.wait(timeout=10)
        self.sink.shutdown()
        self.sink.server_close()
        self.out.close()


def print_notifications(lab):
    for t, name, p in lab.notifications():
        print(render(t, name, p))

Two details are worth knowing now. The harness starts Alertmanager with --cluster.listen-address= and an empty value. The project README says of that flag: “empty string disables HA mode”, and a lone test instance does not need the clustering that high availability uses. And it passes --web.external-url, because without it Alertmanager builds the address it puts in notifications from the machine’s host name, and my first test run printed mine. Now the smallest useful configuration:

route:
  receiver: sink
  group_wait: 2s
receivers:
  - name: sink
    webhook_configs:
      - url: http://127.0.0.1:9099/sink

One route (the root, which every config needs) with one receiver, and a group_wait of two seconds so you do not wait the default thirty. The first script starts the sink and Alertmanager on the real default ports, adds an alert with amtool, waits five seconds and prints the notification:

# s2_first.py
"""s2_first.py: start Alertmanager with first.yml, add one alert with amtool, show the webhook payload."""
import json
import subprocess
import time

from amlab import AMTOOL, Lab, read

lab = Lab("first", read("first.yml"), ports=(9093, 9099), echo=True).start()
cmd = [AMTOOL, "alert", "add", "alertname=DiskFull", "service=checkout", "severity=page",
       '--annotation=summary="Disk is 95 percent full"', "--alertmanager.url=http://127.0.0.1:9093"]
print("$ amtool alert add alertname=DiskFull service=checkout severity=page --annotation='summary=\"Disk is 95 percent full\"' --alertmanager.url=http://127.0.0.1:9093")
p = subprocess.run(cmd, capture_output=True, text=True)
print("exit code", p.returncode, (p.stdout + p.stderr).strip())
time.sleep(5)
t, name, body = lab.notifications()[0]
print(f"the first webhook call arrived {t:.1f} s after the alert was added, at /{name}")
print(json.dumps(body, indent=2))
lab.stop()

Run it with python s2_first.py. Here is what it printed:

$ amtool alert add alertname=DiskFull service=checkout severity=page --annotation='summary="Disk is 95 percent full"' --alertmanager.url=http://127.0.0.1:9093
exit code 0 
  2.0s  sink             firing   group={}
          F  alertname=DiskFull service=checkout severity=page
the first webhook call arrived 2.0 s after the alert was added, at /sink

The first block, with F in it, is receiver.py doing its job as the notification arrives, two seconds after the alert was added. That delay is the group_wait you configured. Here is the JSON itself:

{
  "receiver": "sink",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "DiskFull",
        "service": "checkout",
        "severity": "page"
      },
      "annotations": {
        "summary": "Disk is 95 percent full"
      },
      "startsAt": "2026-11-15T15:08:35.571244-05:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "",
      "fingerprint": "b0cf240b74b4dde1"
    }
  ],
  "notification_reason": "first notification",
  "groupLabels": {},
  "commonLabels": {
    "alertname": "DiskFull",
    "service": "checkout",
    "severity": "page"
  },
  "commonAnnotations": {
    "summary": "Disk is 95 percent full"
  },
  "routeLabels": {},
  "externalURL": "http://localhost:9093",
  "version": "4",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0
}

This is the payload the configuration reference documents under webhook_config, which it introduces with “The Alertmanager will send HTTP POST requests in the following JSON format to the configured endpoint”. Read it from the top. version is the payload format and is "4". status is firing because at least one alert in the group is firing. alerts lists every alert, each with its own status, its labels and annotations, a startsAt and an endsAt. An endsAt of 0001-01-01T00:00:00Z means “not ended”. groupLabels is empty because this configuration has no group_by. commonLabels holds the labels every alert in the group shares. groupKey identifies the group, and notification_reason says why this message was sent; here it is the first one. The fingerprint is the alert’s identity, computed from its labels.

A gotcha with the annotation. My first attempt passed the annotation without quotes, as --annotation=summary=Disk is 95 percent full. The alert was still added and the exit code was 0, but amtool printed a warning that starts “Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible”, and suggested the quoted form summary="Disk is 95 percent full". The configuration reference says the same about the new parser: “make sure all regular expressions and values are double-quoted”. Quote every value that holds spaces or punctuation, and in a POSIX shell wrap the whole argument in single quotes so the inner double quotes survive, as the script above does.

How to know it worked: the sink printed one firing block with your alert about two seconds after the amtool alert add line. If nothing arrives, check that port 9099 is free and that alertmanager.exe started; the harness writes its log to work/first/alertmanager.log.

Step 3: Build the routing tree and test it without sending anything

Now the real configuration. The goal: pages go to the on-call receiver, pages about payments go to the payments team’s own on-call, tickets go to a queue, and anything that does not match is caught by a receiver called triage where a human can look at it and fix the labels. Here is routes.yml:

# routes.yml: pages go to the on-call phone, tickets go to a queue,
# and anything that matches neither lands in triage.
route:
  receiver: triage
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: oncall
      group_wait: 10s
      repeat_interval: 1h
      routes:
        - matchers: [team=~"payments|billing"]
          receiver: payments-oncall
    - matchers: [severity="ticket"]
      receiver: tickets
      repeat_interval: 24h

inhibit_rules:
  - name: page-covers-ticket
    source_matchers: [severity="page"]
    target_matchers: [severity="ticket"]
    equal: [service]

receivers:
  - name: triage
    webhook_configs:
      - url: http://127.0.0.1:9099/triage
  - name: oncall
    webhook_configs:
      - url: http://127.0.0.1:9099/oncall
  - name: payments-oncall
    webhook_configs:
      - url: http://127.0.0.1:9099/payments-oncall
  - name: tickets
    webhook_configs:
      - url: http://127.0.0.1:9099/tickets

Read the route block from the top. The root route has no matchers and names triage as its receiver. The reference says the top-level route “must match all alerts (i.e. not have any configured matchers)”, so anything that matches no child ends up at the root’s own receiver. It also sets group_by: [alertname, service] and three timers. Child routes inherit these: “Most optional configuration parameters (receiver, group_by, group_wait, group_interval, repeat_interval) are inherited from its parent node if not set.” The page route overrides two timers, so pages wait 10 seconds instead of 30 and repeat every hour instead of every four.

Inside the page route sits a child that matches team=~"payments|billing". The =~ operator means “matches this regular expression”. A page about payments takes that child and goes to payments-oncall; any other page stays at oncall. The reference describes the walk: “If continue is set to false, it stops after the first matching child”, and “If an alert does not match any children of a node (no matching child nodes, or none exist), the alert is handled based on the configuration parameters of the current node.”

The inhibit_rules block is explained in step 7. The receivers at the end all POST to the sink on port 9099, each under its own path so you can tell them apart. Now ask amtool to check the file and draw the tree. Neither command needs a running Alertmanager, because they only read the file:

# s3_routes.py
"""s3_routes.py: read the routing tree and ask amtool where a handful of alerts would go."""
import sys

from amlab import amtool

sys.stdout.reconfigure(encoding="utf-8")


def show(cmd):
    out, rc = amtool(*cmd)
    print("$ amtool " + " ".join(cmd))
    print(out)
    print()


show(["check-config", "routes.yml"])
show(["config", "routes", "show", "--config.file=routes.yml"])
CASES = [
    "alertname=PaymentsErrors severity=page service=checkout team=payments",
    "alertname=DiskFull severity=page service=search team=search",
    "alertname=SlowQueries severity=ticket service=search",
    "alertname=DiskFull severity=critical service=search",
    "alertname=DiskFull severity=Page service=search",
    "alertname=NodeDown",
    "alertname=PaymentsErrors severity=page service=checkout team=payments-eu",
]
for labels in CASES:
    show(["config", "routes", "test", "--config.file=routes.yml", *labels.split()])
show(["config", "routes", "test", "--tree", "--config.file=routes.yml", "severity=page", "team=billing"])
show(["config", "routes", "test", "--config.file=continue.yml", "severity=page"])
show(["config", "routes", "test", "--config.file=routes.yml", "--verify.receivers=tickets", "severity=page"])

Run python s3_routes.py. It prints a lot, so I will take the output in pieces. First the syntax check and the tree:

$ amtool check-config routes.yml
Checking 'routes.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 4 receivers
 - 0 templates

$ amtool config routes show --config.file=routes.yml
Routing tree:
.
└── default-route  receiver: triage
    ├── {severity="page"}  receiver: oncall
    │   └── {team=~"payments|billing"}  receiver: payments-oncall
    └── {severity="ticket"}  receiver: tickets

The tree is the quickest way to spot a mistake, because it shows the nesting you intended and the matchers exactly as Alertmanager parsed them. Now seven alerts, each as a command that asks “where would you send an alert with these labels?”:

$ amtool config routes test --config.file=routes.yml alertname=PaymentsErrors severity=page service=checkout team=payments
payments-oncall

$ amtool config routes test --config.file=routes.yml alertname=DiskFull severity=page service=search team=search
oncall

$ amtool config routes test --config.file=routes.yml alertname=SlowQueries severity=ticket service=search
tickets

$ amtool config routes test --config.file=routes.yml alertname=DiskFull severity=critical service=search
triage

$ amtool config routes test --config.file=routes.yml alertname=DiskFull severity=Page service=search
triage

$ amtool config routes test --config.file=routes.yml alertname=NodeDown
triage

$ amtool config routes test --config.file=routes.yml alertname=PaymentsErrors severity=page service=checkout team=payments-eu
oncall

Take these one at a time, because each teaches something.

  • Cases 1 to 3 do what you meant: a payments page goes to payments-oncall, a page from another team to oncall, a ticket to tickets.
  • Case 4 used severity=critical, a word that exists in many companies’ conventions but not in this tree, and it went to triage. Case 5 is severity=Page, with a capital letter, and it went to triage too: matching is case sensitive. Case 6 has no severity label at all. These three are the silent failures. Nothing errors. The alert is delivered, just not to the place you wanted, and a real page ends up in a channel built for leftovers. This is why the catch-all receiver should be somewhere people actually look.
  • Case 7 used team=payments-eu, and it went to oncall, not payments-oncall. The configuration reference explains why: “The regex is anchored on both ends. To un-anchor the regex, use .*<regex>.* .” The pattern payments|billing matches the whole value payments or the whole value billing, never a prefix.

The last three commands show the extras:

$ amtool config routes test --tree --config.file=routes.yml severity=page team=billing
Matching routes:
.
└── default-route
    └── {severity="page"}
        └── {team=~"payments|billing"}  receiver: payments-oncall


payments-oncall

$ amtool config routes test --config.file=continue.yml severity=page
chat-mirror,oncall

$ amtool config routes test --config.file=routes.yml --verify.receivers=tickets severity=page
oncall
WARNING: Expected receivers did not match resolved receivers.

With --tree, amtool prints only the branch the alert took. The next command uses a second file, continue.yml, to show what continue: true changes:

# continue.yml: mirror every page into a chat channel and still page the on-call
route:
  receiver: triage
  routes:
    - matchers: [severity="page"]
      receiver: chat-mirror
      continue: true
    - matchers: [severity="page"]
      receiver: oncall
receivers:
  - name: triage
  - name: chat-mirror
  - name: oncall

The first route matches pages and sends them to chat-mirror, and because of continue: true the alert keeps going down the list and also matches the second route. The test prints both receivers, chat-mirror,oncall, in the order they appear in the file. Use this to copy pages to a chat channel while still paging a person. Finally, --verify.receivers=tickets says “I expect exactly this list”; when the real answer is oncall amtool prints a warning, and the exit status is 1 (I saw 1 when I ran it by hand). That is the hook the CI gate in step 10 builds on.

Common mistakes in this step. A broad route written before a narrower sibling matches first, and without continue: true the narrower route is never reached, so order matters (planted mistake 7 in step 10 shows it). A matcher value that contains spaces or punctuation needs double quotes, or Alertmanager falls back to its old parser and logs a warning like the one in step 2. And amtool config routes test tells you which receiver an alert reaches, but nothing about timing or about inhibition; later steps cover both.

Step 4: Know which labels your rules really produce

A routing tree can only match on the labels an alert actually carries, and they come from two places: the series that the alert’s query returned, and the labels: you write in the rule. One query function breaks the first source in a way that is easy to miss: absent(), the standard way to alert when a metric disappears. The Prometheus documentation shows its result: absent(nonexistent{job="myjob"}) gives {job="myjob"}. There is no series, so there are no labels to copy; absent() can only derive them from the equality matchers in the selector. If your routing depends on a team label that comes from your scrape configuration, an absent() alert will not have it.

You can prove this with promtool from the earlier post, without running Prometheus. Two files: a rule file with a normal alert and an absent() alert, and a unit test that feeds in a series that stops.

groups:
  - name: checkout
    rules:
      - alert: CheckoutErrors
        expr: rate(http_requests_failed_total{job="checkout"}[5m]) > 0.5
        for: 2m
        labels:
          severity: page
          service: checkout
      - alert: CheckoutMetricsMissing
        expr: absent(up{job="checkout"})
        for: 2m
        labels:
          severity: page
          service: checkout
rule_files:
  - absent_rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_failed_total{job="checkout", instance="a:9100", team="payments"}'
        values: '0+60x20'
      - series: 'up{job="checkout", instance="a:9100", team="payments"}'
        values: '1x5 _x20'
    alert_rule_test:
      - eval_time: 8m
        alertname: CheckoutErrors
        exp_alerts:
          - exp_labels:
              severity: page
              service: checkout
              job: checkout
              instance: "a:9100"
              team: payments
      - eval_time: 15m
        alertname: CheckoutMetricsMissing
        exp_alerts:
          - exp_labels:
              severity: page
              service: checkout
              job: checkout

The test supplies two series for the checkout job, both carrying team="payments". The error counter climbs one request per second for the whole test, which is above the 0.5 threshold. The up series has six samples, for minutes 0 to 5, and then nothing (the underscore means “no sample”). At 15 minutes absent() has been true for long enough to pass its two-minute for:. The test lists the labels each alert must have at the moment it is checked, and promtool compares the whole label set, so any label you leave out of the expectation makes the test fail. The script below runs it and then asks amtool where each alert would go, then fixes the rule and runs both again:

# s9_absent.py
"""s9_absent.py: which labels does an absent() alert carry, and where does Alertmanager send it?"""
import difflib
import os
import subprocess

from amlab import HERE, amtool, read

PROMTOOL = os.environ.get("PROMTOOL", "promtool")


def run(args):
    p = subprocess.run([PROMTOOL, *args], capture_output=True, text=True, encoding="utf-8", cwd=HERE)
    return (p.stdout + p.stderr).strip()


def route(labels):
    out, _ = amtool("config", "routes", "test", "--config.file=routes.yml", *labels.split())
    print(f"$ amtool config routes test --config.file=routes.yml {labels}\n{out}")


print("$ promtool test rules absent_test.yml")
print(run(["test", "rules", "absent_test.yml"]))
print()
route("alertname=CheckoutErrors severity=page service=checkout job=checkout instance=a:9100 team=payments")
route("alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout")

wrong = read("absent_test.yml") + "              team: payments\n"
with open(f"{HERE}/absent_test_wrong.yml", "w", newline="\n") as f:
    f.write(wrong)
print("$ promtool test rules absent_test_wrong.yml   (the absent() alert is expected to carry team: payments)")
print("\n".join(line for line in run(["test", "rules", "absent_test_wrong.yml"]).splitlines() if line.strip())[:900])
print()

rules = read("absent_rules.yml")
fixed_rules = rules.replace("          service: checkout\n", "          service: checkout\n          team: payments\n")
test = read("absent_test.yml")
fixed_test = test.replace("absent_rules.yml", "absent_rules_fixed.yml") + "              team: payments\n"
with open(f"{HERE}/absent_rules_fixed.yml", "w", newline="\n") as f:
    f.write(fixed_rules)
with open(f"{HERE}/absent_test_fixed.yml", "w", newline="\n") as f:
    f.write(fixed_test)
print()
print("--- diff absent_rules.yml absent_rules_fixed.yml")
print("".join(difflib.unified_diff(rules.splitlines(1), fixed_rules.splitlines(1), "absent_rules.yml", "absent_rules_fixed.yml", n=1)))
print("--- diff absent_test.yml absent_test_fixed.yml")
print("".join(difflib.unified_diff(test.splitlines(1), fixed_test.splitlines(1), "absent_test.yml", "absent_test_fixed.yml", n=1)))
print("$ promtool test rules absent_test_fixed.yml")
print(run(["test", "rules", "absent_test_fixed.yml"]))
print()
route("alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout team=payments")

Set the environment variable PROMTOOL to the path of promtool (or put it on your PATH) and run python s9_absent.py. I ran it with the 3.16.0 binary. Output:

$ promtool test rules absent_test.yml
Processing test file: absent_test.yml
  SUCCESS

$ amtool config routes test --config.file=routes.yml alertname=CheckoutErrors severity=page service=checkout job=checkout instance=a:9100 team=payments
payments-oncall
$ amtool config routes test --config.file=routes.yml alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout
oncall
$ promtool test rules absent_test_wrong.yml   (the absent() alert is expected to carry team: payments)
Processing test file: absent_test_wrong.yml
  FAILED:
    alertname: CheckoutMetricsMissing, time: 15m, 
        exp:[
            0:
              Labels:{alertname="CheckoutMetricsMissing", job="checkout", service="checkout", severity="page", team="payments"}
              Annotations:{}
            ], 
        got:[
            0:
              Labels:{alertname="CheckoutMetricsMissing", job="checkout", service="checkout", severity="page"}
              Annotations:{}
            ]


--- diff absent_rules.yml absent_rules_fixed.yml
--- absent_rules.yml
+++ absent_rules_fixed.yml
@@ -9,2 +9,3 @@
           service: checkout
+          team: payments
       - alert: CheckoutMetricsMissing
@@ -15 +16,2 @@
           service: checkout
+          team: payments

--- diff absent_test.yml absent_test_fixed.yml
--- absent_test.yml
+++ absent_test_fixed.yml
@@ -1,3 +1,3 @@
 rule_files:
-  - absent_rules.yml
+  - absent_rules_fixed.yml
 evaluation_interval: 1m
@@ -27 +27,2 @@
               job: checkout
+              team: payments

$ promtool test rules absent_test_fixed.yml
Processing test file: absent_test_fixed.yml
  SUCCESS

$ amtool config routes test --config.file=routes.yml alertname=CheckoutMetricsMissing severity=page service=checkout job=checkout team=payments
payments-oncall

Read it in order. The first test passes: it says the absent() alert has severity, service and job and no team, and promtool agrees. The two routing lines show the consequence. The error alert, which has team=payments, goes to payments-oncall. The missing-metrics alert for the same service goes to plain oncall, and the payments team never hears about it. The third block is the proof that the missing label is real, not my assumption: when the test was told that the absent() alert should carry team: payments, promtool failed it and printed the expected and the actual label sets one above the other, and the actual set has no team.

The fix is to put every label that routing needs into the rule’s labels:, and not to rely on labels borrowed from series. The first diff shows the two added lines in the rule file, the second diff the added expectation, the fixed test passes, and the route test now prints payments-oncall for the absent() alert as well.

Rule of thumb: for every label your routing and inhibition rules use, ask where it comes from. If the answer is “the target”, an absent() alert will not have it. Write routing labels into the rule.

Step 5: Control how many notifications you get with group_by

When ten pods of one service all fail, you want one message that says “ten pods are failing”, not ten messages. That is grouping, and group_by decides it: alerts with identical values for the listed labels share a group, and each group sends one notification. The experiment below posts the same six alerts under five different group_by settings, each with its own Alertmanager running side by side. Three are HighLatency for the checkout service (pods p1 to p3), two are HighLatency for search (p1 and p2), and one is DiskFull for checkout.

# s4_group.py
"""s4_group.py: the same six alerts under five group_by settings."""
import threading

from amlab import Lab

TEMPLATE = """route:
  receiver: sink
  group_wait: 2s
{group_by}
receivers:
  - name: sink
    webhook_configs:
      - url: http://127.0.0.1:9099/sink
"""
VARIANTS = [
    "  group_by: [alertname, service]",
    "  group_by: [alertname, service, pod]",
    "  group_by: [alertname, service, cluster]",
    "  group_by: ['...']",
    "  # no group_by line at all",
]
ALERTS = ([dict(alertname="HighLatency", service="checkout", pod=f"p{i}") for i in (1, 2, 3)]
          + [dict(alertname="HighLatency", service="search", pod=f"p{i}") for i in (1, 2)]
          + [dict(alertname="DiskFull", service="checkout", pod="p1")])
results = {}


def run(k, line):
    lab = Lab(f"group{k}", TEMPLATE.format(group_by=line), k=k).start()
    lab.post(*ALERTS)
    lab.wait_until(6)
    results[k] = [len(p["alerts"]) for _, _, p in lab.notifications()]
    lab.stop()


threads = [threading.Thread(target=run, args=(k, line)) for k, line in enumerate(VARIANTS)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print(f"{'group_by setting':<40}{'notifications':>14}   alerts in each")
for k, line in enumerate(VARIANTS):
    sizes = sorted(results[k], reverse=True)
    print(f"{line.strip():<40}{len(sizes):>14}   {', '.join(map(str, sizes))}")

Run python s4_group.py:

group_by setting                         notifications   alerts in each
group_by: [alertname, service]                       3   3, 2, 1
group_by: [alertname, service, pod]                  6   1, 1, 1, 1, 1, 1
group_by: [alertname, service, cluster]              3   3, 2, 1
group_by: ['...']                                    6   1, 1, 1, 1, 1, 1
# no group_by line at all                            1   6
  • [alertname, service] gives three notifications of three, two and one alerts. This is usually the right choice: one message per kind of problem per service, listing every pod in it.
  • Adding pod gives six notifications, one per alert. You have grouped by something so specific that grouping does nothing.
  • Listing a label that no alert has, here cluster, changes nothing: still three notifications. A label that is missing counts as an empty value, the same for every alert, so it does not split the groups and it does not raise an error. A typo in a group_by label name therefore fails quietly in the direction of fewer, larger groups.
  • ['...'] is the special value that means group by every label, which the reference says “effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want”. The result is six notifications.
  • With no group_by at all, every alert in the route lands in one group. One notification carries all six alerts. That can be right for a small system and wrong for a large one, because a single group also shares one set of timers.

Pick group_by labels that name what a human would act on: the kind of problem and the thing that owns it. Pod and instance names belong in the notification’s list of alerts, not in the group key.

Step 6: Understand the three timers by watching a clock

Grouping needs time to work: Alertmanager has to wait a moment so related alerts can arrive together, and it has to decide how often to speak again. Three settings control that, and the reference describes each of them.

  • group_wait: “How long to wait before sending the first notification for a new group of alerts.” It exists so that alerts from other rule groups, and any alert that should inhibit this one, can arrive first. The reference adds: “If an alert is resolved before group_wait has elapsed, no notification will be sent for that alert. This reduces noise of flapping alerts.”
  • group_interval: how long to wait before sending news about a group that has already been notified. It is “a recurring timer that starts as soon as group_wait has elapsed”. At each tick Alertmanager sends a notification if alerts were added or resolved since the last one.
  • repeat_interval: how long to wait before repeating a notification when nothing has changed. “Since the repeat_interval is checked after each group_interval, it should be a multiple of the group_interval. If it’s not, the repeat_interval is rounded up to the next multiple of the group_interval.”

The defaults are 30 seconds, 5 minutes and 4 hours, which makes experiments slow. The harness therefore shrinks every timer in routes.yml with the FAST table you saw in amlab.py: 30 seconds becomes 2, 10 seconds becomes 1, 5 minutes becomes 5 seconds, and the three repeat intervals become 60, 30 and 60 seconds. The experiment below sets its own timers directly: group_wait: 2s, group_interval: 5s, repeat_interval: 12s. Note that 12 is not a multiple of 5, on purpose. The script posts and resolves alerts at fixed times, then prints your actions and Alertmanager’s notifications on one timeline, with >> marking what the script did:

# s5_timers.py
"""s5_timers.py: watch group_wait, group_interval and repeat_interval on a clock."""
from datetime import datetime, timedelta, timezone

from amlab import Lab
from receiver import render

CONFIG = """route:
  receiver: sink
  group_by: [alertname, service]
  group_wait: 2s
  group_interval: 5s
  repeat_interval: 12s
receivers:
  - name: sink
    webhook_configs:
      - url: http://127.0.0.1:9099/sink
"""
A = dict(alertname="HighLatency", service="checkout", pod="p1")
B = dict(alertname="HighLatency", service="checkout", pod="p2")
FLAP = dict(alertname="HighLatency", service="search", pod="p1")
OLD = dict(alertname="HighLatency", service="billing", pod="p1")

lab = Lab("timers", CONFIG).start()
log = []


def at(t, text, action):
    lab.wait_until(t)
    action()
    log.append((lab.now(), f"{lab.now():5.1f}s  >> {text}"))


at(0, "post A (checkout p1)", lambda: lab.post(A))
at(3, "post B (checkout p2)", lambda: lab.post(B))
at(10, "post FLAP (search p1)", lambda: lab.post(FLAP))
at(11, "resolve FLAP, one second later", lambda: lab.resolve(FLAP))
at(14, "post OLD (billing p1) with startsAt 60 s in the past",
   lambda: lab.post(OLD, started=datetime.now(timezone.utc) - timedelta(seconds=60)))
at(16, "resolve OLD", lambda: lab.resolve(OLD))
at(26, "resolve A", lambda: lab.resolve(A))
at(33, "resolve B", lambda: lab.resolve(B))
lab.wait_until(44)
rows = log + [(t, render(t, name, p)) for t, name, p in lab.notifications()]
for _, text in sorted(rows, key=lambda r: r[0]):
    print(text)
lab.stop()

Run python s5_timers.py. It takes 44 seconds:

  0.0s  >> post A (checkout p1)
  2.0s  sink             firing   group={alertname=HighLatency service=checkout}
          F  alertname=HighLatency pod=p1 service=checkout
  3.0s  >> post B (checkout p2)
  7.0s  sink             firing   group={alertname=HighLatency service=checkout}
          F  alertname=HighLatency pod=p1 service=checkout
          F  alertname=HighLatency pod=p2 service=checkout
 10.0s  >> post FLAP (search p1)
 11.0s  >> resolve FLAP, one second later
 14.0s  >> post OLD (billing p1) with startsAt 60 s in the past
 14.0s  sink             firing   group={alertname=HighLatency service=billing}
          F  alertname=HighLatency pod=p1 service=billing
 16.0s  >> resolve OLD
 19.0s  sink             resolved group={alertname=HighLatency service=billing}
          R  alertname=HighLatency pod=p1 service=billing
 22.0s  sink             firing   group={alertname=HighLatency service=checkout}
          F  alertname=HighLatency pod=p1 service=checkout
          F  alertname=HighLatency pod=p2 service=checkout
 26.0s  >> resolve A
 27.0s  sink             firing   group={alertname=HighLatency service=checkout}
          R  alertname=HighLatency pod=p1 service=checkout
          F  alertname=HighLatency pod=p2 service=checkout
 33.0s  >> resolve B
 37.0s  sink             resolved group={alertname=HighLatency service=checkout}
          R  alertname=HighLatency pod=p2 service=checkout

Read the timeline row by row.

  • 0.0 to 2.0 seconds. Alert A is posted and the first notification arrives two seconds later: that is group_wait.
  • 3.0 to 7.0. Alert B joins the same group one second after the first notification went out. It does not get its own two second wait. It waits for the next tick of the group’s recurring timer, which is 7.0 (the first flush at 2.0 plus five seconds), and then goes out together with A.
  • 10.0 and 11.0. A search alert fires and resolves one second later, before its group_wait could expire. No notification is ever sent for it, which is the flapping protection the reference describes. Look for the search group in the output: it is not there.
  • 14.0. This alert was posted with a start time 60 seconds in the past, and the notification came out at 14.0, with no two second wait. In the source of the dispatcher at the v0.35.0 tag, line 605 checks alrt.StartsAt.Add(ag.opts.GroupWait).Before(now) when a group is created and logs “Alert is old enough for immediate flush, resetting timer to zero”. So group_wait is counted from the alert’s own start time. An alert that has already been firing for longer than group_wait when its group is created is sent at once. The configuration reference does not mention this; I found it by measuring and confirmed it in the source.
  • 16.0 and 19.0. That alert is resolved at 16.0, and a resolved notification arrives at 19.0, the next tick after 14.0. Resolutions travel on the same ticks as new alerts.
  • 22.0. The checkout group is notified again although nothing changed. The last message went out at 7.0 and repeat_interval is 12 seconds, but Alertmanager only looks at ticks: at 12.0 it had been 5 seconds, at 17.0 it had been 10, at 22.0 it had been 15, so it repeated at 22.0. A twelve second setting behaved as fifteen. That is the rounding the reference describes.
  • 26.0 and 27.0. A is resolved at 26.0 and the next tick, 27.0, sends the group with R for A and F for B. The group’s status is still firing because B is.
  • 33.0 and 37.0. B is resolved just after the tick at 32.0, so the final resolved message waits for 37.0.

What this means for your own settings. Keep group_wait short on the routes that page (10 seconds in routes.yml), but not so short that an inhibiting alert from another rule group has no time to arrive; step 7 shows what happens when it is late. Choose repeat_interval as a multiple of group_interval, so the number you write is the number you get. And remember that a resolved message is sent on a tick, so “the page cleared” can reach the receiver up to one group_interval after the alert actually ended.

Step 7: Stop tickets from piling on a page with inhibition

When a service is down, you do not also want a ticket for its slow queries and a ticket for its high latency. The page already says everything. An inhibition rule expresses that. The one in routes.yml reads:

inhibit_rules:
  - name: page-covers-ticket
    source_matchers: [severity="page"]
    target_matchers: [severity="ticket"]
    equal: [service]

In words: while an alert matching the source matchers is firing, mute alerts matching the target matchers, but only when both alerts have the same value for every label in equal. So a page about service=checkout mutes tickets about checkout and leaves tickets about other services alone. The reference gives two rules that matter here. The first: “Semantically, a missing label and a label with an empty value are the same thing. Therefore, if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply.” The second concerns an alert that matches both sides: such an alert “cannot be inhibited by alerts for which the same is true (including itself)”, but the reference adds “we recommend to choose target and source matchers in a way that alerts never match both sides”, as this rule does.

The experiment runs the same eight alerts through two Alertmanagers, one with the equal line and one without. Three pages (P1, P0, P4) and five tickets (T0 to T4) are involved. P1 is a page for checkout and T1 a ticket for checkout, so T1 should be inhibited. T2 and T3 are tickets for search and billing and should get through. P0 and T0 are a page and a ticket that have no service label at all. T4 is a ticket for orders that arrives first, and P4, the page for orders, arrives 4.5 seconds later, after T4’s notification has already gone out. At 7.0 seconds the script reads Alertmanager’s own view of every alert, and at 8.5 it resolves P1.

# s6_inhibit.py
"""s6_inhibit.py: one inhibit rule, with and without its equal line."""
import threading
import time

from amlab import Lab, fast, print_notifications, read, state_line

P1 = dict(alertname="DiskFull", severity="page", service="checkout")
T1 = dict(alertname="SlowQueries", severity="ticket", service="checkout")  # same service as P1
T2 = dict(alertname="SlowQueries", severity="ticket", service="search")    # another service
T3 = dict(alertname="CertExpiry", severity="ticket", service="billing")    # another service
P0 = dict(alertname="NodeDown", severity="page")                           # no service label
T0 = dict(alertname="NodeDiskWarn", severity="ticket")                     # no service label
T4 = dict(alertname="SlowQueries", severity="ticket", service="orders")    # the ticket arrives first
P4 = dict(alertname="DiskFull", severity="page", service="orders")         # the page arrives 4.5 s later

base = fast(read("routes.yml"))
configs = [("with equal: [service]", base),
           ("without the equal line", base.replace("    equal: [service]\n", ""))]
assert configs[0][1] != configs[1][1]
labs = [Lab(f"inhibit{k}", cfg, k=k).start() for k, (_, cfg) in enumerate(configs)]
states = {}


def run(lab):
    lab.t0 = lab.sink.t0 = time.monotonic()
    lab.post(P1, T1, T2, T3, P0, T0, T4)
    lab.wait_until(4.5)
    lab.post(P4)
    lab.wait_until(7)
    rows = sorted(lab.alerts(), key=lambda a: (a["labels"]["alertname"], a["labels"].get("service", "")))
    states[lab.name] = [state_line(a) for a in rows]
    lab.wait_until(8.5)
    lab.resolve(P1)
    lab.wait_until(16)


threads = [threading.Thread(target=run, args=(lab,)) for lab in labs]
for t in threads:
    t.start()
for t in threads:
    t.join()
for lab, (title, _) in zip(labs, configs):
    print(f"=== {title}")
    print("t=0.0s  posted P1 T1 T2 T3 P0 T0 T4 together; t=4.5s posted P4; t=7.0s read /api/v2/alerts; t=8.5s resolved P1")
    print_notifications(lab)
    print("--- /api/v2/alerts at t=7.0s")
    for line in states[lab.name]:
        print(line)
    print()
    lab.stop()

Run python s6_inhibit.py. First the Alertmanager with the equal line:

t=0.0s  posted P1 T1 T2 T3 P0 T0 T4 together; t=4.5s posted P4; t=7.0s read /api/v2/alerts; t=8.5s resolved P1
  1.0s  oncall           firing   group={alertname=NodeDown}
          F  alertname=NodeDown severity=page
  1.0s  oncall           firing   group={alertname=DiskFull service=checkout}
          F  alertname=DiskFull service=checkout severity=page
  2.0s  tickets          firing   group={alertname=SlowQueries service=orders}
          F  alertname=SlowQueries service=orders severity=ticket
  2.0s  tickets          firing   group={alertname=CertExpiry service=billing}
          F  alertname=CertExpiry service=billing severity=ticket
  2.0s  tickets          firing   group={alertname=SlowQueries service=search}
          F  alertname=SlowQueries service=search severity=ticket
  5.5s  oncall           firing   group={alertname=DiskFull service=orders}
          F  alertname=DiskFull service=orders severity=page
 11.0s  oncall           resolved group={alertname=DiskFull service=checkout}
          R  alertname=DiskFull service=checkout severity=page
 12.0s  tickets          firing   group={alertname=SlowQueries service=checkout}
          F  alertname=SlowQueries service=checkout severity=ticket
--- /api/v2/alerts at t=7.0s
CertExpiry   service=billing severity=ticket           active     
DiskFull     service=checkout severity=page            active     
DiskFull     service=orders severity=page              active     
NodeDiskWarn severity=ticket                           suppressed inhibited
NodeDown     severity=page                             active     
SlowQueries  service=checkout severity=ticket          suppressed inhibited
SlowQueries  service=orders severity=ticket            suppressed inhibited
SlowQueries  service=search severity=ticket            active     
  • T1 is held back and T2 and T3 get through. At 2.0 seconds the ticket queue receives only the search, billing and orders tickets. The checkout ticket T1 is missing. In the API view at 7.0 seconds it has state suppressed with reason inhibited.
  • Two alerts with no service label inhibit each other. T0 (NodeDiskWarn) never reached the queue, because P0 (NodeDown) has no service either, and a missing label equals an empty one. That is exactly the case the reference describes. Sometimes it is what you want. When it is not, give every alert a service label in its rule, as step 4 recommends.
  • Order matters: T4 was sent before P4 arrived. The orders ticket went out at 2.0, because no page for orders existed yet. P4 arrived at 4.5 and from then on T4 shows as suppressed and inhibited in the API, but the message that already left is not recalled. This is why the reference ties group_wait to inhibition: it is long enough “for one or more inhibiting alerts to arrive and mute any target alerts before the first notification”. If your page alert can fire a minute after the ticket alert, the ticket will be sent.
  • Inhibition ends when the source resolves. P1 was resolved at 8.5 seconds, and the held-back checkout ticket was delivered at 12.0, the next tick of its group. Until then it waited quietly; the page’s own resolved message arrived at 11.0.

Now the same alerts when the rule has no equal line:

t=0.0s  posted P1 T1 T2 T3 P0 T0 T4 together; t=4.5s posted P4; t=7.0s read /api/v2/alerts; t=8.5s resolved P1
  1.0s  oncall           firing   group={alertname=NodeDown}
          F  alertname=NodeDown severity=page
  1.0s  oncall           firing   group={alertname=DiskFull service=checkout}
          F  alertname=DiskFull service=checkout severity=page
  5.5s  oncall           firing   group={alertname=DiskFull service=orders}
          F  alertname=DiskFull service=orders severity=page
 11.0s  oncall           resolved group={alertname=DiskFull service=checkout}
          R  alertname=DiskFull service=checkout severity=page
--- /api/v2/alerts at t=7.0s
CertExpiry   service=billing severity=ticket           suppressed inhibited
DiskFull     service=checkout severity=page            active     
DiskFull     service=orders severity=page              active     
NodeDiskWarn severity=ticket                           suppressed inhibited
NodeDown     severity=page                             active     
SlowQueries  service=checkout severity=ticket          suppressed inhibited
SlowQueries  service=orders severity=ticket            suppressed inhibited
SlowQueries  service=search severity=ticket            suppressed inhibited

With no equal, the rule says “while any page is firing anywhere, mute every ticket”. All five tickets are suppressed, nothing at all reaches the queue, and no error or warning tells you so. In production this is one of the nastiest inhibition bugs, because the pager works and the tickets silently stop whenever any page is open. Always write the equal line, and use labels that every alert carries.

Step 8: Mute alerts on purpose with silences

A silence is the manual version of inhibition: a matcher, a start, an end, an author and a comment. You use it for planned work, so a maintenance window does not generate tickets. The reference says alerts are checked “whether they match all the equality or regular expression matchers of an active silence. If they do, no notifications will be sent out for that alert.” You can create silences in the web interface or with amtool. The script below uses amtool for every silence command, and it prints the commands as you would type them against the default port; the lab itself runs on another port. Silence IDs are random, so the script replaces them with <id>.

# s7_silence.py
"""s7_silence.py: silence an alert with amtool, test an unanchored regex, expire the silence."""
from amlab import Lab, amtool, fast, mask, print_notifications, read, state_line

lab = Lab("silence", fast(read("routes.yml"))).start()
URL = f"--alertmanager.url={lab.url}"
SHOWN = "--alertmanager.url=http://127.0.0.1:9093"
T1 = dict(alertname="SlowQueries", severity="ticket", service="checkout")
T2 = dict(alertname="SlowQueries", severity="ticket", service="search")
T3 = dict(alertname="CertExpiry", severity="ticket", service="billing")
T5 = dict(alertname="DiskWarn", severity="ticket", service="billing")


def run(*args):
    out, rc = amtool(*args, URL)
    print("$ amtool " + mask(" ".join(f"'{a}'" if " " in a or "~" in a else a for a in args)) + " " + SHOWN)
    print(mask(out))
    return out


def silence(*matchers, comment):
    out = run("silence", "add", *matchers, f"--comment={comment}", "--author=lab", "--duration=10m")
    return out.splitlines()[-1].strip()


def states(title):
    print(f"--- /api/v2/alerts, {title}")
    for a in sorted(lab.alerts(), key=lambda a: (a["labels"]["alertname"], a["labels"]["service"])):
        print(state_line(a))


sid = silence("service=checkout", "severity=ticket", comment="planned database maintenance")
lab.post(T1, T2)
lab.wait_until(4)
states(f"t={lab.now():.1f}s")
run("silence", "query")
silence('service=~"bill"', comment="regex without a wildcard")
lab.post(T3)
lab.wait_until(8)
states(f"t={lab.now():.1f}s, after silencing service=~\"bill\" and posting CertExpiry for billing")
silence('service=~"bill.*"', comment="regex with a wildcard")
lab.post(T5)
lab.wait_until(12)
states(f"t={lab.now():.1f}s, after silencing service=~\"bill.*\" and posting DiskWarn for billing")
t_expire = lab.now()
run("silence", "expire", sid)
lab.wait_until(20)
print(f"--- webhook calls (silence for checkout expired at t={t_expire:.1f}s)")
print_notifications(lab)
lab.stop()

Run python s7_silence.py. It takes 20 seconds:

$ amtool silence add service=checkout severity=ticket '--comment=planned database maintenance' --author=lab --duration=10m --alertmanager.url=http://127.0.0.1:9093
<id>
--- /api/v2/alerts, t=4.0s
SlowQueries  service=checkout severity=ticket          suppressed silenced
SlowQueries  service=search severity=ticket            active     
$ amtool silence query --alertmanager.url=http://127.0.0.1:9093
ID                                    Matchers                              Ends At                  Created By  Comment                       
<id>  service="checkout" severity="ticket"  2026-11-15 20:18:35 UTC  lab         planned database maintenance
$ amtool silence add 'service=~"bill"' '--comment=regex without a wildcard' --author=lab --duration=10m --alertmanager.url=http://127.0.0.1:9093
<id>
--- /api/v2/alerts, t=8.0s, after silencing service=~"bill" and posting CertExpiry for billing
CertExpiry   service=billing severity=ticket           active     
SlowQueries  service=checkout severity=ticket          suppressed silenced
SlowQueries  service=search severity=ticket            active     
$ amtool silence add 'service=~"bill.*"' '--comment=regex with a wildcard' --author=lab --duration=10m --alertmanager.url=http://127.0.0.1:9093
<id>
--- /api/v2/alerts, t=12.0s, after silencing service=~"bill.*" and posting DiskWarn for billing
CertExpiry   service=billing severity=ticket           suppressed silenced
DiskWarn     service=billing severity=ticket           suppressed silenced
SlowQueries  service=checkout severity=ticket          suppressed silenced
SlowQueries  service=search severity=ticket            active     
$ amtool silence expire <id> --alertmanager.url=http://127.0.0.1:9093

--- webhook calls (silence for checkout expired at t=12.0s)
  2.0s  tickets          firing   group={alertname=SlowQueries service=search}
          F  alertname=SlowQueries service=search severity=ticket
  6.1s  tickets          firing   group={alertname=CertExpiry service=billing}
          F  alertname=CertExpiry service=billing severity=ticket
 17.0s  tickets          firing   group={alertname=SlowQueries service=checkout}
          F  alertname=SlowQueries service=checkout severity=ticket
  • The exact silence works. After silence add service=checkout severity=ticket, the checkout ticket is suppressed with reason silenced and the search ticket, which the silence does not match, is delivered at 2.0 seconds. amtool silence query lists the silence with its matchers, end time, author and comment.
  • A regular expression without a wildcard silences nothing. service=~"bill" was meant to catch the billing service, but the regular expression is anchored on both ends, so it only matches the value bill. The billing ticket CertExpiry stayed active and its notification went out at 6.1 seconds. Only service=~"bill.*" matched, and it silenced both billing alerts, including the one that had already been notified. A silence stops future notifications; it does not recall one that was sent.
  • Expiring a silence does not deliver at once. The silence on checkout was expired at 12.0 seconds and the held-back ticket arrived at 17.0: the next tick of its group, five seconds later.

Good habits that follow from this: match on the narrowest labels that describe the work (service and severity here, never alertname alone), always give an author and a comment so the next person knows why the alert is quiet, and set a duration so it expires by itself. Check the result in the alert list; a silence that matches nothing looks exactly like one that works until the alert fires.

Step 9: Keep tickets quiet outside working hours with time intervals

A route can also be muted by the clock. You define named time_intervals and attach them to routes with mute_time_intervals (do not notify during these times) or active_time_intervals (notify only during these times). The reference says of muting: “When a route is muted it will not send any notifications, but otherwise acts normally (including ending the route-matching process if the continue option is not set.)” It also contains a sentence worth testing: “mute_time_intervals is not inherited from the parent route. If omitted, the route has no mute time intervals, even if the parent specifies some.” That is unlike the timers and group_by, which children do inherit.

The experiment mutes the whole ticket route all day (an interval from 00:00 to 24:00, so the result does not depend on the time you run it) and adds a child route for team=db that does not mention muting. Three alerts go in: a page, a ticket for search, and a ticket for the db team.

# s8_mute.py
"""s8_mute.py: mute_time_intervals on a route, and the fact that a child route does not inherit them."""
import os
import sys
import urllib.request

from amlab import HERE, Lab, amtool, print_notifications, state_line

sys.stdout.reconfigure(encoding="utf-8")
CONFIG = """route:
  receiver: triage
  group_by: [alertname, service]
  group_wait: 2s
  group_interval: 5s
  routes:
    - matchers: [severity="page"]
      receiver: oncall
    - matchers: [severity="ticket"]
      receiver: tickets
      mute_time_intervals: [all-day]
      routes:
        - matchers: [team="db"]
          receiver: db-tickets
time_intervals:
  - name: all-day
    time_intervals:
      - times:
          - start_time: '00:00'
            end_time: '24:00'
receivers:
  - name: triage
    webhook_configs:
      - url: http://127.0.0.1:9099/triage
  - name: oncall
    webhook_configs:
      - url: http://127.0.0.1:9099/oncall
  - name: tickets
    webhook_configs:
      - url: http://127.0.0.1:9099/tickets
  - name: db-tickets
    webhook_configs:
      - url: http://127.0.0.1:9099/db-tickets
"""
PAGE = dict(alertname="DiskFull", severity="page", service="checkout")
TICKET = dict(alertname="SlowQueries", severity="ticket", service="search")
DB_TICKET = dict(alertname="SlowQueries", severity="ticket", service="orders", team="db")
lab = Lab("mute", CONFIG).start()
lab.post(PAGE, TICKET, DB_TICKET)
lab.wait_until(7)
print("--- webhook calls")
print_notifications(lab)
print("--- /api/v2/alerts at t=7.0s")
for a in sorted(lab.alerts(), key=lambda a: (a["labels"]["alertname"], a["labels"]["service"])):
    print(state_line(a))
lab.stop()
print()
out, rc = amtool("check-config", "hours.yml")
print("$ amtool check-config hours.yml")
print(out)
ZIP = f"{HERE}/zoneinfo.zip"
if not os.path.exists(ZIP):
    urllib.request.urlretrieve("https://raw.githubusercontent.com/golang/go/master/lib/time/zoneinfo.zip", ZIP)
out, rc = amtool("check-config", "hours.yml", env={"ZONEINFO": ZIP})
print()
print("$ ZONEINFO=zoneinfo.zip amtool check-config hours.yml")
print(out)

The second half of the script checks a realistic working-hours file, hours.yml, which uses active_time_intervals and a time zone:

# hours.yml: tickets reach the queue only during working hours
route:
  receiver: triage
  routes:
    - matchers: [severity="ticket"]
      receiver: tickets
      active_time_intervals: [working-hours]
time_intervals:
  - name: working-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'
        location: 'America/New_York'
receivers:
  - name: triage
  - name: tickets

Run python s8_mute.py:

--- webhook calls
  2.0s  db-tickets       firing   group={alertname=SlowQueries service=orders}
          F  alertname=SlowQueries service=orders severity=ticket team=db
  2.0s  oncall           firing   group={alertname=DiskFull service=checkout}
          F  alertname=DiskFull service=checkout severity=page
--- /api/v2/alerts at t=7.0s
DiskFull     service=checkout severity=page            active     
SlowQueries  service=orders severity=ticket team=db    active     
SlowQueries  service=search severity=ticket            active     

$ amtool check-config hours.yml
Checking 'hours.yml'  FAILED: unknown time zone America/New_York (on Windows platforms, you may have to pass the time zone database using the ZONEINFO environment variable, see https://pkg.go.dev/time#LoadLocation for details)

amtool.exe: error: failed to validate 1 file(s)

$ ZONEINFO=zoneinfo.zip amtool check-config hours.yml
Checking 'hours.yml'  SUCCESS
Found:
 - global config
 - route
 - 0 inhibit rules
 - 2 receivers
 - 0 templates
  • The muted route sends nothing, and a child route is not muted with it. The page was delivered, the db team’s ticket was delivered through the child route db-tickets, and the search ticket matched the muted tickets route and was not sent. The documentation’s sentence about inheritance held.
  • The API still calls the muted alert active. In the alert list the search ticket looks like any other firing alert. A time mute stops the notification but, in this run, did not show up as suppressed, so an alert that is “active” in the list is not proof that anyone was told. Check the receiver, not the list.
  • Windows needs a time zone database for location. The first check of hours.yml failed with “unknown time zone America/New_York (on Windows platforms, you may have to pass the time zone database using the ZONEINFO environment variable”. Go’s documentation for LoadLocation says it looks at “the directory or uncompressed zip file named by the ZONEINFO environment variable”. The script downloaded zoneinfo.zip from the Go repository (a stored, uncompressed zip of about 400 KB), pointed ZONEINFO at it, and the same check then printed SUCCESS. On Linux and macOS Go reads the system time zone database, so no extra step is normally needed. When you run the real Alertmanager on Windows, set the same variable in its environment.

I did not test what happens to an alert that fires while its route is muted once the muting interval ends, so check that with your own clock before you rely on “tickets arrive at 09:00”.

Step 10: Put the routing tree under a CI gate and test the gate itself

Everything above can be checked by hand, and every configuration change deserves the same checks without anyone remembering to do them. The gate has two parts. amtool check-config catches files that Alertmanager would refuse to load. Then a table of cases, each a set of labels and the receiver you expect, goes through amtool config routes test. Any difference fails the build.

# gate.py
"""gate.py: fail unless the routing tree sends every test case to the receiver we expect.

usage: python gate.py alertmanager.yml
"""
import sys

from amlab import amtool

CASES = [
    ("payments page", "alertname=PaymentsErrors severity=page service=checkout team=payments", "payments-oncall"),
    ("billing page", "alertname=PaymentsErrors severity=page service=checkout team=billing", "payments-oncall"),
    ("other team page", "alertname=DiskFull severity=page service=search team=search", "oncall"),
    ("page without team", "alertname=NodeDown severity=page", "oncall"),
    ("ticket", "alertname=SlowQueries severity=ticket service=search", "tickets"),
    ("severity typo falls to triage", "alertname=DiskFull severity=critical service=search", "triage"),
    ("no severity falls to triage", "alertname=NodeDown", "triage"),
    ("team prefix is not a match", "alertname=PaymentsErrors severity=page team=payments-eu", "oncall"),
]


def check(config, cwd="."):
    out, rc = amtool("check-config", config, cwd=cwd)
    if rc != 0:
        line = next((x for x in out.splitlines() if "FAILED" in x or "rror" in x), out.splitlines()[0])
        return ["check-config failed: " + line.strip()]
    problems = []
    for name, labels, want in CASES:
        out, rc = amtool("config", "routes", "test", f"--config.file={config}", *labels.split(), cwd=cwd)
        if out.strip() != want:
            problems.append(f"{name}: expected {want}, got {out.strip()}")
    return problems


if __name__ == "__main__":
    problems = check(sys.argv[1])
    print("\n".join(problems) or f"all {len(CASES)} routing cases pass")
    sys.exit(1 if problems else 0)

Run python gate.py routes.yml. The expected output is one line:

all 8 routing cases pass

A test that cannot fail is worthless, so the next script plants fourteen mistakes in routes.yml, one at a time, and runs the gate against each. Eleven are things people really do: a typo in a matcher value, a regular expression written as a prefix, a catch-all route placed first, a misspelled receiver. Three are changes that the gate has no way to see, and I included them to measure that blind spot honestly.

# mutants.py
"""mutants.py: plant mistakes in routes.yml one at a time and see which ones gate.py catches."""
import os

from amlab import HERE, read
from gate import check

BASE = read("routes.yml")
PAGE_ROUTE = '    - matchers: [severity="page"]\n'
MUTANTS = [
    ("page matcher typo", '- matchers: [severity="page"]', '- matchers: [severity="pager"]'),
    ("ticket matcher typo", '- matchers: [severity="ticket"]', '- matchers: [severity="tickets"]'),
    ("regex loses billing", 'team=~"payments|billing"', 'team=~"payments"'),
    ("regex written as a prefix", 'team=~"payments|billing"', 'team=~"pay|billing"'),
    ("pages go to the ticket queue", "receiver: oncall", "receiver: tickets"),
    ("payments child sends to oncall", "receiver: payments-oncall", "receiver: oncall"),
    ("catch-all route placed first", "  routes:\n" + PAGE_ROUTE, "  routes:\n    - receiver: tickets\n" + PAGE_ROUTE),
    ("receiver name misspelled", "receiver: oncall", "receiver: oncal"),
    ("duration without a unit", "group_wait: 10s", "group_wait: 10"),
    ("root receiver is the ticket queue", "route:\n  receiver: triage", "route:\n  receiver: tickets"),
    ("page route loses its child", '      routes:\n        - matchers: [team=~"payments|billing"]\n          receiver: payments-oncall\n', ""),
    ("page repeats every second", "repeat_interval: 1h", "repeat_interval: 1s"),
    ("inhibit rule loses its equal line", "    equal: [service]\n", ""),
    ("grouping switched off", "group_by: [alertname, service]", "group_by: ['...']"),
]
os.makedirs(f"{HERE}/work/mutants", exist_ok=True)
caught = 0
print(f"{'#':>2}  {'planted mistake':<36}{'verdict':<10}what the gate said")
for n, (name, old, new) in enumerate(MUTANTS, 1):
    assert old in BASE, name
    path = f"{HERE}/work/mutants/m{n:02d}.yml"
    with open(path, "w", newline="\n", encoding="utf-8") as f:
        f.write(BASE.replace(old, new, 1))
    problems = check(f"m{n:02d}.yml", cwd=f"{HERE}/work/mutants")
    caught += bool(problems)
    said = problems[0] if len(problems) == 1 else (f"{len(problems)} cases fail, first: {problems[0]}" if problems else "")
    print(f"{n:>2}  {name:<36}{'caught' if problems else 'SURVIVED':<10}{said}")
print(f"\nthe unmodified file: {check('routes.yml', cwd=HERE) or 'passes'}")
print(f"caught {caught} of {len(MUTANTS)}")

Run python mutants.py:

 #  planted mistake                     verdict   what the gate said
 1  page matcher typo                   caught    5 cases fail, first: payments page: expected payments-oncall, got triage
 2  ticket matcher typo                 caught    ticket: expected tickets, got triage
 3  regex loses billing                 caught    billing page: expected payments-oncall, got oncall
 4  regex written as a prefix           caught    payments page: expected payments-oncall, got oncall
 5  pages go to the ticket queue        caught    3 cases fail, first: other team page: expected oncall, got tickets
 6  payments child sends to oncall      caught    2 cases fail, first: payments page: expected payments-oncall, got oncall
 7  catch-all route placed first        caught    7 cases fail, first: payments page: expected payments-oncall, got tickets
 8  receiver name misspelled            caught    check-config failed: Checking 'm08.yml'  FAILED: undefined receiver "oncal" used in route
 9  duration without a unit             caught    check-config failed: Checking 'm09.yml'  FAILED: not a valid duration string: "10"
10  root receiver is the ticket queue   caught    2 cases fail, first: severity typo falls to triage: expected triage, got tickets
11  page route loses its child          caught    2 cases fail, first: payments page: expected payments-oncall, got oncall
12  page repeats every second           SURVIVED  
13  inhibit rule loses its equal line   SURVIVED  
14  grouping switched off               SURVIVED  

the unmodified file: passes
caught 11 of 14
  • Nine mistakes were caught by the routing cases. A misspelled severity value fails five cases, a catch-all placed first fails seven (it swallows every alert before the real routes see it), and a regular expression written as the prefix pay fails the payments case.
  • Two were caught by check-config alone. A receiver name with a letter missing is rejected with “undefined receiver”, and a group_wait without a unit with “not a valid duration string”. Those files cannot even load, so the routing cases never run.
  • Three survived. A page route that repeats every second, an inhibit rule without its equal line, and grouping switched off all pass both checks, because amtool config routes test only reports which receiver an alert reaches. It says nothing about timers, grouping or inhibition. The inhibit change is the dangerous one, since step 7 showed it silencing every ticket.

What would catch the three survivors? Experiments like steps 5 to 7: start a throwaway Alertmanager with the candidate configuration, post a fixed set of alerts, and assert on the notifications that reach the sink. That takes about fifteen to twenty seconds per scenario (the inhibition experiment above ran for 16 seconds), which is acceptable in CI for a file this important. A cheaper partial check is to compare the timers and the inhibit_rules block against a reviewed copy and require a human approval when they change.

To use the gate in a pipeline, keep gate.py and its case table in the repository next to the configuration, and run bin/amtool check-config alertmanager.yml and python gate.py alertmanager.yml in a job that must pass before a merge. The script exits with status 1 and prints one line per failing case.

Common mistakes, in one place

  • A label spelled differently from the route’s matcher (critical against page, Page against page) sends the alert to the catch-all route, quietly. Make the catch-all receiver one that people read, and add cases to your gate for each label value you rely on.
  • A regular expression matcher is anchored on both ends. team=~"pay" does not match payments, in routes or in silences.
  • absent() alerts do not carry the labels of the missing series. Write the labels your routing needs into the rule.
  • A group_by label that no alert has changes nothing and raises no error; ['...'] turns grouping off.
  • A repeat_interval that is not a multiple of group_interval is rounded up.
  • An inhibit rule with no equal line mutes every target alert while any source alert is firing. Alerts missing every equal label inhibit each other.
  • An inhibiting alert that arrives after group_wait is too late to stop the first notification, and nothing recalls it.
  • A silence does not recall a notification that already went out; it only stops later ones.
  • A muted route still shows its alerts as active in the alert list, and a child route does not inherit its parent’s mute intervals.

Confirm that everything works end to end

Run the steps in order from one folder that holds bin, the Python files and the YAML files, and compare what you see with this post. python s2_first.py should print one firing notification after about 2 seconds. python s3_routes.py should print SUCCESS for the file, the tree, and the receivers in the cases above. python s4_group.py should print the 3, 6, 3, 6 and 1 notification counts. python s5_timers.py should show notifications at about 2, 7, 14, 19, 22, 27 and 37 seconds; the clock readings can differ by a tenth of a second on your machine. python s6_inhibit.py and python s7_silence.py should show the same suppressed and delivered alerts. python gate.py routes.yml should print “all 8 routing cases pass”, and python mutants.py should end with “caught 11 of 14”. The IDs and wall-clock timestamps in the JSON will differ; the receivers, group labels and counts should not.

Where to go next

  • Connect a real Prometheus. Prometheus finds Alertmanager through the alerting: section of its configuration. The overview page warns: “It’s important not to load balance traffic between Prometheus and its Alertmanagers, but instead, point Prometheus to a list of all Alertmanagers.” If you run more than one instance, turn the clustering that this lab disabled back on; the README lists the --cluster.* flags.
  • Replace the sink with a real receiver (PagerDuty, Slack, Opsgenie and others have their own settings), keep the credentials out of the repository, and read the post on verifying webhook signatures if your own service receives these webhooks.
  • Add a heartbeat. Create an alert that always fires and route it to an external service that complains when it stops arriving. It is the only way to learn that the pipeline itself has died.
  • Write notification templates so each message states the service, the runbook link and what to do first.
  • Test the alert rules too. The earlier SLO post shows how to unit-test rules with promtool, so that both halves, “does the alert fire” and “does it reach the right person”, are covered by tests.

Sources and further reading

  • Alertmanager configuration reference: routes, timers, inhibition, time intervals, matchers and the webhook payload.
  • Alertmanager overview: grouping, inhibition, silences and high availability.
  • Alertmanager v0.35.0 release and the README for the amtool examples and the cluster flags.
  • dispatch.go at v0.35.0, the immediate-flush check for old alerts.
  • Prometheus query functions: absent().
  • Go time.LoadLocation, which explains ZONEINFO, and the zoneinfo.zip in the Go repository.

Tags:

AlertingAlertmanageramtoolObservabilityOn-CallPrometheus

Share

A brass nail with a threaded screw section lying on graph paper, a fastener that is both nail and screw
Previous Post

Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses

A stack of chocolate chip cookies on a white plate beside a cup of tea
Next Post

NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
16 Nov
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
16 Nov
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
Trending
November 16, 2026
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 15, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
November 15, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
November 15, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query

Related Posts

A laptop wrapped in a chain and padlock, illustrating least-privilege controls for AI agents.
Learning Hub

How to Secure Tool-Using AI Agents Before They Touch Production

June 8, 2026
Colorful sticky notes arranged on an office wall, symbolizing governance checklists and planning.
Learning Hub

AI Governance for Agentic Apps: A Practical Checklist for Builders

June 8, 2026
A technician connects green fiber optic cables at a data center, representing a private production inference endpoint.
Learning Hub

How to Deploy a Fine-Tuned LLM Behind a Private Production Inference Endpoint

June 8, 2026
Narrow aisle behind black supercomputer racks in a data center
Learning Hub

Kubernetes SELinux Volume Labeling: What Cluster Operators Should Audit Before v1.37

June 8, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026