TRENDING
A black manual telephone switchboard with rows of orange jacks and red patch cords plugged into the lower row, above a rotary dial and a handset
November 16, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
A brass nail with a threaded screw section lying on graph paper, a fastener that is both nail and screw
November 16, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
An amber warning beacon on a white surface, a flashing-light stand-in for the FBI’s FLASH alert
November 16, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query
Two eggs and a colour-changing egg timer in a saucepan of water on an induction hob
November 16, 2026
How to Set gRPC Deadlines and Retries in Python, Pass Them Down a Call Chain, and Stop Retries From Charging a Card Twice
A whole lemon and a halved lemon side by side on a black background
November 16, 2026
IBM’s LLMON Turns Prompt-Injection Defense Into a Token Layout, but the “Nearly Zero” Chart Is One Benchmark, the Paper’s Own Red Team Lets 15% to 20% of Indirect Attacks Through, and the Shipped Template Lets a Document Type the Delimiters
16 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
A manual paper cutter with a wooden grid board, a clear plastic guard and a long cutting arm, a picture of text being trimmed to fit
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
A brown llama and her gray cria walking across a salt flat beside a red lake in Bolivia
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
A stack of chocolate chip cookies on a white plate beside a cup of tea
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 16, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 302 Posts
News 307 Posts
Learning Hub 274 Posts
Home/Learning Hub/How to Alert on Your SLO Without Paging for Noise: Write Multi-Window Burn-Rate Alerts in Prometheus and Unit-Test Them With promtool
Learning Hub

How to Alert on Your SLO Without Paging for Noise: Write Multi-Window Burn-Rate Alerts in Prometheus and Unit-Test Them With promtool

Learn what an error budget and a burn rate are, write multi-window burn-rate alerts in Prometheus, and prove with promtool unit tests and replayed traffic when each alert fires, then plant nine...

November 14, 2026 26 Min Read
19

Alerts fail in two opposite ways. Either nobody hears about a real problem until customers complain, or an alert wakes someone at 3 a.m. for a blip that cost almost nothing. In this tutorial you will build alert rules for an HTTP API that avoid both. They fire when the service is spending its monthly allowance of failures too fast, they page a person only when waiting would be expensive, and they open a quiet ticket when the leak is slow. You will also prove how they behave: you will replay five kinds of traffic through the rules, write unit tests that say exactly when each alert should and should not fire, and plant nine mistakes in the rules to check that your tests notice every one.

Table Of Content

  • What you will build
  • The five ideas you need first
  • Why one threshold is not enough
  • Prerequisites
  • Step 1: Install promtool and check that it runs
  • Step 2: Do the error budget arithmetic
  • Step 3: Write the alert most people write first
  • Step 4: Write the burn-rate alerts
  • Step 5: Build a lab that replays traffic through the rules
  • Step 6: Replay the traffic and read the results
  • What the usual alert does
  • What the burn-rate alerts do
  • Step 7: Check promtool against an independent model
  • Step 8: Turn the replay into unit tests
  • Step 9: Break the rules on purpose
  • Common mistakes and how to avoid them
  • Putting recording rules and alerts in different groups
  • Adding for to a page alert without thinking
  • A range window with too few samples
  • Testing only the loud cases
  • A budget typed for the wrong SLO
  • Low-traffic services
  • Silence when the metric disappears
  • Confirm everything works end to end
  • Where to go next
  • Sources

Nothing here needs a running Prometheus server, an Alertmanager, a Kubernetes cluster or a paid account. The one new tool is promtool, the command line program that ships with Prometheus. You do not need earlier experience with it. If you have written a Prometheus query before you are ready, and if you have not, every rule is explained line by line.

The alert design is the multiwindow, multi-burn-rate alert from chapter 5 of the Google SRE Workbook. What this post adds is the testing: the workbook gives you the rules, and here you check them against traffic you can reproduce on your own machine.

What you will build

  • a rule file with three alerts, SLOBurnFast, SLOBurnSlow and SLOBurnTicket, built on recording rules;
  • a small Python lab (standard library only) that creates traffic, replays it through promtool and reports when each alert fires;
  • an independent Python model of the same alerts, to cross-check promtool;
  • a unit test file, slo_test.yml, that you can run in CI;
  • a check that nine realistic mistakes in the rule file are all caught by those tests.

The five ideas you need first

Five words carry this whole topic. Read them once and the rest will make sense.

  • SLI (service level indicator). A measurement of how well the service is doing. Here it is the share of requests that fail with an HTTP 5xx status code.
  • SLO (service level objective). The target for that measurement over a period. Here it is 99.9 percent of requests succeeding over 30 days.
  • Error budget. What the SLO lets you get wrong: 100 percent minus 99.9 percent, so 0.1 percent of requests may fail. Spread over 30 days that is the same as 43.2 minutes of total outage.
  • Burn rate. How fast you are spending the budget compared with spending it evenly. A burn rate of 1 uses exactly the whole budget in 30 days. A burn rate of 14.4 uses it in about two days.
  • Window. The stretch of recent time an alert looks at, for example the last hour. Long windows are steady but slow to notice. Short windows notice quickly but are jumpy.

Two Prometheus words complete the picture. A recording rule saves the result of a query as a new time series, so alerts can reuse it. An alerting rule is a query plus labels and text, and the alert fires while the query returns something.

Why one threshold is not enough

The alert most people write first says “page me when the error ratio over the last five minutes is above the SLO’s failure rate”. It has one speed setting. A tiny blip trips it, a slow leak that will drain the budget in a week trips it just as loudly, and nothing in it tells you how much budget has actually gone.

The workbook’s answer is to alert on the burn rate and to look through two windows at once. The long window says that a meaningful slice of budget has gone. The short window says that the problem is still happening right now, so the alert stops soon after the problem does. The workbook also gives a rule of thumb: “make the short window 1/12 the duration of the long window”. Its Table 5-8, “Recommended parameters for a 99.9% SLO alerting configuration”, is where we start:

Severity Long window Short window Burn rate Budget consumed
Page 1 hour 5 minutes 14.4 2%
Page 6 hours 30 minutes 6 5%
Ticket 3 days 6 hours 1 10%

A page interrupts a person at once. A ticket goes into a queue to be handled in working hours. The three rows are the three alerts you will write.

Prerequisites

  • Operating system: tested on Windows 11 with Git Bash. The Linux and macOS archives from the same release page contain the same promtool, but I did not test them.
  • Python: 3.13 (tested with 3.13.14). The lab uses only the standard library.
  • promtool: from the Prometheus 3.16.0 release (the version this tutorial was tested with). The download is the full Prometheus archive of about 117 MB, and you only extract one program from it.
  • Prior knowledge: you know what an HTTP status code is and you have seen a counter such as http_requests_total with a code label. If your application names things differently, change the metric and label names in the rule text.
  • Time: about 40 minutes, most of it reading.

Step 1: Install promtool and check that it runs

Prometheus releases ship promtool inside the same archive as the server. You will download the archive for your platform, check its checksum, and pull out the one program you need. Run these commands in Git Bash (or any POSIX shell):

mkdir slo-lab && cd slo-lab
curl -L -O https://github.com/prometheus/prometheus/releases/download/v3.16.0/prometheus-3.16.0.windows-amd64.zip
curl -L -O https://github.com/prometheus/prometheus/releases/download/v3.16.0/sha256sums.txt
grep "windows-amd64.zip" sha256sums.txt
sha256sum prometheus-3.16.0.windows-amd64.zip

The last two commands print the checksum the release publishes and the checksum of the file you downloaded. They must be identical:

89c03c2f24400fb58668440b6d7226f68564acda27d657e77acfbda42be32cb3 *prometheus-3.16.0.windows-amd64.zip
89c03c2f24400fb58668440b6d7226f68564acda27d657e77acfbda42be32cb3  prometheus-3.16.0.windows-amd64.zip

A matching checksum tells you the download was not corrupted on the way. It does not prove the release page itself is trustworthy, because both numbers come from the same place. Now extract promtool and tell the lab where it is:

python -c "import zipfile; zipfile.ZipFile('prometheus-3.16.0.windows-amd64.zip').extract('prometheus-3.16.0.windows-amd64/promtool.exe', '.')"
export PROMTOOL="$(cygpath -m "$PWD")/prometheus-3.16.0.windows-amd64/promtool.exe"
"$PROMTOOL" --version | head -1

The last command prints the first line of the version output:

promtool, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d)

If you see a version number, you are set. On Windows, cygpath -m turns the Git Bash path into the C:/... form that Python understands. On Linux or macOS, unpack the Linux archive instead and use export PROMTOOL="$PWD/prometheus-3.16.0.linux-amd64/promtool" (I did not test that variant).

Step 2: Do the error budget arithmetic

Before writing any rule, check the numbers in the workbook table yourself. This takes a few lines of Python and removes the mystery from “14.4”. Save this as slo_math.py:

# slo_math.py
"""Error budget and burn-rate arithmetic for a 99.9 percent SLO over 30 days."""
SLO = 0.999
PERIOD_HOURS = 30 * 24
budget = 1 - SLO

print(f"SLO {SLO * 100:g}% over 30 days: error budget {budget * 100:.3f}% of requests,"
      f" the same as {budget * PERIOD_HOURS * 60:.1f} minutes of total outage")
print()
print(f"{'burn rate':>9}  {'window':>6}  {'threshold ratio':>15}  {'days to empty':>13}  {'budget used':>11}")
for burn, hours in ((14.4, 1), (6, 6), (1, 72)):
    days = PERIOD_HOURS / burn / 24
    consumed = burn * hours / PERIOD_HOURS
    print(f"{burn:>9}  {str(hours) + ' h':>6}  {burn * budget * 100:>14.4f}%  {days:>13.2f}  {consumed * 100:>10.1f}%")

Run it with python slo_math.py:

SLO 99.9% over 30 days: error budget 0.100% of requests, the same as 43.2 minutes of total outage

burn rate  window  threshold ratio  days to empty  budget used
     14.4     1 h          1.4400%           2.08         2.0%
        6     6 h          0.6000%           5.00         5.0%
        1    72 h          0.1000%          30.00        10.0%

Read the columns from left to right. The threshold ratio is the burn rate times the budget: a burn rate of 14.4 on a 0.1 percent budget means 1.44 percent of requests failing. Days to empty is 30 days divided by the burn rate, so a burn of 14.4 empties the whole budget in about 2.08 days. Budget used is the burn rate times the window length divided by the 720 hours in 30 days, which is the share of the budget used up if the burn lasts the whole window: 2, 5 and 10 percent, the same percentages as the workbook table. That is also where the odd numbers come from. You pick how much budget you are willing to lose before someone is told, 2 percent in an hour for example, and solve for the burn rate: 0.02 times 720 hours, divided by 1 hour, is 14.4.

Step 3: Write the alert most people write first

You will measure the usual alert next to the better ones, so write it down. Create a folder named rules and save this as rules/naive.yml:

groups:
  - name: naive
    interval: 1m
    rules:
      - record: job:naive_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[5m]))
      - record: job:naive_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[1h]))
      - alert: HighErrorRate
        expr: job:naive_errors_per_request:ratio_rate5m > 0.001
        labels:
          severity: page
      - alert: BurnFast1hOnly
        expr: job:naive_errors_per_request:ratio_rate1h > (14.4 * 0.001)
        labels:
          severity: page

The file has one rule group that is evaluated every minute. The two record rules turn counters into a ratio. rate(http_requests_total{...}[5m]) is the per-second increase of each counter over the last five minutes, code=~"5.." keeps the series whose status code starts with 5, and sum by (job) adds the series together so that one number per job remains. Dividing failures by all requests gives the share of requests that failed, a number between 0 and 1. The first alert, HighErrorRate, is the common one: page when more than 0.1 percent of requests failed over five minutes. The second, BurnFast1hOnly, uses only the one-hour window at the 14.4 threshold. It is here to show what the short window buys you later.

Step 4: Write the burn-rate alerts

Now the real rules. Save this as rules/slo.yml:

groups:
  - name: slo-burn-rate
    interval: 1m
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[5m]))
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[30m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[1h]))
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[6h]))
      - record: job:slo_errors_per_request:ratio_rate3d
        expr: |
          sum by (job) (rate(http_requests_total{job="api",code=~"5.."}[3d]))
          /
          sum by (job) (rate(http_requests_total{job="api"}[3d]))
      - alert: SLOBurnFast
        expr: |
          job:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)"
      - alert: SLOBurnSlow
        expr: |
          job:slo_errors_per_request:ratio_rate6h > (6 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "api is burning its 30-day error budget at 6x or faster (6h and 30m windows)"
      - alert: SLOBurnTicket
        expr: |
          job:slo_errors_per_request:ratio_rate3d > (1 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate6h > (1 * 0.001)
        labels:
          severity: ticket
        annotations:
          summary: "api is burning its 30-day error budget at 1x or faster (3d and 6h windows)"

Three details are worth understanding before you move on.

  • Five ratios, three alerts. The five recording rules compute the failure ratio over 5 minutes, 30 minutes, 1 hour, 6 hours and 3 days. The names follow the workbook’s pattern, job:slo_errors_per_request:ratio_rate1h. Each alert compares a long-window ratio and a short-window ratio with the same threshold and requires both, joined by and. The thresholds are written as 14.4 * 0.001, burn rate times budget, so a reader can see the intent.
  • Everything is in one group. The Prometheus documentation says “Rules within a group are run sequentially at a regular interval, with the same evaluation time.” Putting the recording rules above the alerts that use them in the same group means each alert sees ratios computed at the same moment.
  • Labels and text. severity: page or severity: ticket is what an Alertmanager route would use to decide between waking someone up and filing a ticket. Setting up Alertmanager is outside this tutorial.

Check that both files are valid with "$PROMTOOL" check rules rules/naive.yml rules/slo.yml:

Checking rules/naive.yml
  SUCCESS: 4 rules found

Checking rules/slo.yml
  SUCCESS: 8 rules found

The count is the number of rules in each file: 2 recording and 2 alerting rules in the first, 5 recording and 3 alerting rules in the second. check rules checks that the files are valid. It cannot tell you whether the thresholds are sensible, and that is what the rest of the tutorial is for.

Step 5: Build a lab that replays traffic through the rules

Real traffic is never the same twice, so to test alerts you need traffic you can repeat. promtool test rules lets you describe counters by hand in a test file and then asks Prometheus’ rule engine what the alerts would have done. The docs explain the notation for the numbers: a+bxn means “series starts at a, then n further samples incrementing by b”. So 0+5999x4319 is a counter that starts at 0 and grows by 5,999 every minute, 4,320 samples in total, which is three days at one sample a minute.

The lab replays five scenarios. Every scenario starts with three days of healthy traffic, so that all windows, even the three-day one, are full when the incident begins, and ends with eight healthy hours so you can watch the alerts reset. Healthy traffic is 6,000 requests a minute (100 a second) with one failing request a minute (code 500, about 0.017 percent). During an incident the failing requests are code 503, which makes the code=~"5.." match matter.

Scenario What happens Why it is in the lab
outage15 Every request fails for 15 minutes A real outage that should page
blip 30 percent of requests fail for one minute A hiccup that should not wake anyone
burn5x 0.5 percent fail for two days A slow leak, about a 5x burn
burn7x 0.7 percent fail for 12 hours A faster leak, about a 7x burn
oneerror One request a minute, and exactly one of them fails A very quiet service

Two files do the work. genrules.py writes the two rule files from Steps 3 and 4 (so you can regenerate them) and can also produce the nine broken variants used in Step 9. Save it first and run python genrules.py; it prints wrote rules/slo.yml and rules/naive.yml.

# genrules.py
"""Generate the Prometheus rule files used in this tutorial (standard library only).

python genrules.py            writes rules/naive.yml and rules/slo.yml
Mistakes for the planted-mistake check are produced with slo_rules(mut="name").
"""
import pathlib

HERE = pathlib.Path(__file__).resolve().parent

# name, severity, burn rate, long window, short window (Table 5-8 of the Google SRE Workbook, chapter 5)
ALERTS = [
    ("SLOBurnFast", "page", "14.4", "1h", "5m"),
    ("SLOBurnSlow", "page", "6", "6h", "30m"),
    ("SLOBurnTicket", "ticket", "1", "3d", "6h"),
]
WINDOWS = ["5m", "30m", "1h", "6h", "3d"]
SUMMARY = {
    "SLOBurnFast": "api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)",
    "SLOBurnSlow": "api is burning its 30-day error budget at 6x or faster (6h and 30m windows)",
    "SLOBurnTicket": "api is burning its 30-day error budget at 1x or faster (3d and 6h windows)",
}

MUTANTS = [
    "factor_low",       # 14.4 typed as 1.44
    "budget_99",        # error budget written for a 99 percent SLO (0.01) instead of 0.001
    "or_not_and",       # long OR short window instead of AND
    "short_eq_long",    # short window copy-pasted from the long one
    "wrong_severity",   # fast page alert labelled as a ticket
    "only_500",         # errors counted as code="500" instead of code=~"5.."
    "rate_5s",          # 5m recording rule written with a 5s range
    "wrong_cmp",        # ticket comparison flipped to <
    "for_15m",          # fast page alert delayed by for: 15m
]


def ratio(window, err, win_text=None):
    w = win_text or window
    return (f'sum by (job) (rate(http_requests_total{{job="api",{err}}}[{w}]))\n'
            f'          /\n'
            f'          sum by (job) (rate(http_requests_total{{job="api"}}[{w}]))')


def slo_rules(mut=None):
    err = 'code="500"' if mut == "only_500" else 'code=~"5.."'
    out = ["groups:", "  - name: slo-burn-rate", "    interval: 1m", "    rules:"]
    for w in WINDOWS:
        wt = "5s" if (mut == "rate_5s" and w == "5m") else None
        out += [f"      - record: job:slo_errors_per_request:ratio_rate{w}",
                "        expr: |",
                f"          {ratio(w, err, wt)}"]
    budget = "0.01" if mut == "budget_99" else "0.001"
    for name, sev, burn, long_w, short_w in ALERTS:
        if name == "SLOBurnFast":
            if mut == "factor_low":
                burn = "1.44"
            if mut == "short_eq_long":
                short_w = long_w
            if mut == "wrong_severity":
                sev = "ticket"
        op = "or" if (mut == "or_not_and" and name == "SLOBurnFast") else "and"
        cmp_ = "<" if (mut == "wrong_cmp" and name == "SLOBurnTicket") else ">"
        out += [f"      - alert: {name}",
                "        expr: |",
                f"          job:slo_errors_per_request:ratio_rate{long_w} {cmp_} ({burn} * {budget})",
                f"          {op}",
                f"          job:slo_errors_per_request:ratio_rate{short_w} {cmp_} ({burn} * {budget})"]
        if mut == "for_15m" and name == "SLOBurnFast":
            out.append("        for: 15m")
        out += ["        labels:", f"          severity: {sev}",
                "        annotations:", f'          summary: "{SUMMARY[name]}"']
    return "\n".join(out) + "\n"


def naive_rules():
    out = ["groups:", "  - name: naive", "    interval: 1m", "    rules:"]
    for w in ("5m", "1h"):
        out += [f"      - record: job:naive_errors_per_request:ratio_rate{w}",
                "        expr: |", f"          {ratio(w, 'code=~\"5..\"')}"]
    out += ["      - alert: HighErrorRate",
            "        expr: job:naive_errors_per_request:ratio_rate5m > 0.001",
            "        labels:", "          severity: page",
            "      - alert: BurnFast1hOnly",
            "        expr: job:naive_errors_per_request:ratio_rate1h > (14.4 * 0.001)",
            "        labels:", "          severity: page"]
    return "\n".join(out) + "\n"


if __name__ == "__main__":
    (HERE / "rules").mkdir(exist_ok=True)
    (HERE / "rules" / "slo.yml").write_text(slo_rules(), encoding="utf-8", newline="\n")
    (HERE / "rules" / "naive.yml").write_text(naive_rules(), encoding="utf-8", newline="\n")
    print("wrote rules/slo.yml and rules/naive.yml")

The second file, lab.py, is the lab itself. Its main parts are these. build() turns a scenario into three counters, one for each status code we use (200, 500 and 503). compact() writes a list of numbers in the a+bxn notation. model() is an independent Python version of the alert logic, used in Step 7. run_matrix() asks promtool a clever question: for every minute of the replay it writes a check that says “no alert should be firing” (exp_alerts: []). Whenever an alert is actually firing, promtool reports a failed check and prints the alerts it got, so the list of failures is exactly the timeline of when each alert fired. suite_text() and cmd_mutants() belong to Steps 8 and 9.

# lab.py
"""SLO burn-rate alert lab: traffic scenarios, a pure-Python model, promtool runs, planted mistakes.

python lab.py matrix    run every scenario through promtool and print when each alert fires
python lab.py suite     write slo_test.yml (a readable unit test) and run it on the clean rules
python lab.py mutants   run slo_test.yml against nine planted mistakes
Needs promtool (set PROMTOOL=<path> if it is not on PATH). Standard library only.
"""
import json
import os
import pathlib
import re
import shutil
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor

import genrules

HERE = pathlib.Path(__file__).resolve().parent
PROMTOOL = os.environ.get("PROMTOOL", "promtool")
WARM = 4320   # three days of healthy history, one sample a minute
TAIL = 480    # eight healthy hours after the incident
SLO_ALERTS = ["SLOBurnFast", "SLOBurnSlow", "SLOBurnTicket"]
ALL_ALERTS = ["HighErrorRate", "BurnFast1hOnly"] + SLO_ALERTS
WINDOW_MIN = {"5m": 5, "30m": 30, "1h": 60, "6h": 360, "3d": 4320}

SCENARIOS = {
    # per_min: requests a minute; frac: failing share during the incident; minutes: incident length;
    # base: baseline failures a minute (code 500); fixed: exact failures a minute during the incident
    "outage15": dict(per_min=6000, frac=1.0, minutes=15, base=1),
    "blip": dict(per_min=6000, frac=0.30, minutes=1, base=1),
    "burn5x": dict(per_min=6000, frac=0.005, minutes=2880, base=1),
    "burn7x": dict(per_min=6000, frac=0.007, minutes=720, base=1),
    "oneerror": dict(per_min=1, frac=0.0, minutes=1, base=0, fixed=1),
}


def build(s):
    """Cumulative counters, one sample a minute: sample i holds the traffic of minute i."""
    n = WARM + s["minutes"] + TAIL
    T = s["per_min"]
    ok, e500, e503 = [0] * n, [0] * n, [0] * n
    for i in range(1, n):
        inc = WARM <= i < WARM + s["minutes"]
        x = (s["fixed"] if s.get("fixed") is not None else round(T * s["frac"])) if inc else 0
        b = s["base"] if x < T else 0
        e503[i] = e503[i - 1] + x
        e500[i] = e500[i - 1] + b
        ok[i] = ok[i - 1] + T - x - b
    return n, ok, e500, e503


def fmt(v):
    return str(int(v))


def compact(vals):
    """Turn a list of numbers into promtool's a+bxn notation (n+1 samples starting at a)."""
    out, i, n = [], 0, len(vals)
    while i < n:
        if i + 1 < n:
            b = vals[i + 1] - vals[i]
            j = i + 1
            while j + 1 < n and vals[j + 1] - vals[j] == b:
                j += 1
            if j - i >= 2:
                out.append(f"{fmt(vals[i])}+{fmt(b)}x{j - i}")
                i = j + 1
                continue
        out.append(fmt(vals[i]))
        i += 1
    return " ".join(out)


def input_block(ok, e500, e503):
    lines = []
    for code, v in (("200", ok), ("500", e500), ("503", e503)):
        lines.append(f"      - series: 'http_requests_total{{job=\"api\",code=\"{code}\"}}'")
        lines.append(f"        values: '{compact(v)}'")
    return lines


def ratio_series(n, ok, e500, e503, window):
    """Failing share over the last `window` minutes, with rate()'s left-open range: window - 1 intervals."""
    k = window - 1
    out = []
    for t in range(n):
        a = max(0, t - k)
        err = (e500[t] + e503[t]) - (e500[a] + e503[a])
        tot = (ok[t] + e500[t] + e503[t]) - (ok[a] + e500[a] + e503[a])
        out.append(err / tot if tot > 0 else None)
    return out


def model(s):
    n, ok, e500, e503 = build(s)
    R = {w: ratio_series(n, ok, e500, e503, m) for w, m in WINDOW_MIN.items()}

    def gt(x, thr):
        return x is not None and x > thr

    fire = {a: [False] * n for a in ALL_ALERTS}
    for t in range(n):
        fire["HighErrorRate"][t] = gt(R["5m"][t], 0.001)
        fire["BurnFast1hOnly"][t] = gt(R["1h"][t], 14.4 * 0.001)
        fire["SLOBurnFast"][t] = gt(R["1h"][t], 14.4 * 0.001) and gt(R["5m"][t], 14.4 * 0.001)
        fire["SLOBurnSlow"][t] = gt(R["6h"][t], 6 * 0.001) and gt(R["30m"][t], 6 * 0.001)
        fire["SLOBurnTicket"][t] = gt(R["3d"][t], 1 * 0.001) and gt(R["6h"][t], 1 * 0.001)
    return fire


DUR = re.compile(r"(\d+)(ms|y|w|d|h|m|s)")
UNIT = {"y": 525600, "w": 10080, "d": 1440, "h": 60, "m": 1, "s": 1 / 60}


def minutes(txt):
    return int(round(sum(int(a) * UNIT[u] for a, u in DUR.findall(txt) if u != "ms")))


def run_promtool(args, cwd):
    p = subprocess.run([PROMTOOL] + args, cwd=str(cwd), capture_output=True, text=True,
                       encoding="utf-8", errors="replace", timeout=1500)
    return p.returncode, p.stdout + p.stderr


def parse_failures(text):
    """Return [(alertname, minute, fired)] for every failed check in promtool's output."""
    res = []
    for chunk in re.split(r"alertname: ", text)[1:]:
        m = re.match(r"(\w+), time: (\S+?),", chunk)
        if not m:
            continue
        got = chunk.split("got:", 1)[1] if "got:" in chunk else ""
        res.append((m.group(1), minutes(m.group(2)), "Labels:" in got))
    return res


def run_matrix(name):
    s = SCENARIOS[name]
    n, ok, e500, e503 = build(s)
    d = HERE / "work" / name
    d.mkdir(parents=True, exist_ok=True)
    out = ["rule_files:", "  - ../../rules/naive.yml", "  - ../../rules/slo.yml", "evaluation_interval: 1m",
           "tests:", "  - interval: 1m", "    input_series:"] + input_block(ok, e500, e503) + ["    alert_rule_test:"]
    for t in range(60, n):
        for a in ALL_ALERTS:
            out += [f"      - eval_time: {t}m", f"        alertname: {a}", "        exp_alerts: []"]
    (d / "matrix.yml").write_text("\n".join(out) + "\n", encoding="utf-8", newline="\n")
    rc, text = run_promtool(["test", "rules", "matrix.yml"], d)
    fired = {a: set() for a in ALL_ALERTS}
    for a, t, f in parse_failures(text):
        if f:
            fired[a].add(t)
    return name, rc, fired


def cmd_matrix():
    with ThreadPoolExecutor(max_workers=5) as ex:
        results = list(ex.map(run_matrix, list(SCENARIOS)))
    rows, worst = [], 0
    print(f"{'scenario':<9} {'alert':<15} {'first fire':>10} {'budget used':>11} {'minutes':>8}  model first fire")
    print(f"{'':<9} {'':<15} {'(min in)':>10} {'(percent)':>11} {'firing':>8}")
    for name, rc, fired in results:
        s = SCENARIOS[name]
        n, ok, e500, e503 = build(s)
        budget_events = 0.001 * s["per_min"] * 43200
        fire = model(s)
        for a in ALL_ALERTS:
            pt = sorted(fired[a])
            mt = [t for t in range(n) if fire[a][t] and t >= 60]
            first = pt[0] - WARM if pt else None
            mfirst = mt[0] - WARM if mt else None
            used = round(100 * e503[pt[0]] / budget_events, 1) if pt else None
            diff = (first - mfirst) if (first is not None and mfirst is not None) else None
            worst = max(worst, abs(diff or 0))
            rows.append(dict(scenario=name, alert=a, first=first, budget_used=used, minutes_firing=len(pt),
                             model_first=mfirst, model_minutes=len(mt), same_minutes=(set(pt) == set(mt)),
                             exit_code=rc))
            f = "never" if first is None else f"{first:+d}"
            u = "-" if used is None else f"{used:.1f}"
            mf = "never" if mfirst is None else f"{mfirst:+d}"
            print(f"{name:<9} {a:<15} {f:>10} {u:>11} {len(pt):>8}  {mf}"
                  f"{'' if set(pt) == set(mt) else '  (minute sets differ)'}")
        print()
    same = sum(r["same_minutes"] for r in rows)
    print(f"model and promtool fire in exactly the same minutes for {same} of {len(rows)} scenario/alert pairs;"
          f" largest first-fire difference {worst} minute(s)")
    os.makedirs(HERE / "out", exist_ok=True)
    (HERE / "out" / "matrix.json").write_text(json.dumps(rows, indent=1), encoding="utf-8")


SUITE_SCENARIOS = ["outage15", "blip", "burn7x", "oneerror"]


def suite_text(rules_path="rules/slo.yml"):
    out = ["rule_files:", f"  - {rules_path}", "evaluation_interval: 1m", "tests:"]
    for name in SUITE_SCENARIOS:
        s = SCENARIOS[name]
        n, ok, e500, e503 = build(s)
        fire = model(s)
        sets = [frozenset(a for a in SLO_ALERTS if fire[a][t]) for t in range(n)]
        times = {WARM - 1, WARM, WARM + 1, WARM + 10}   # also pin the quiet minutes around the incident start
        for t in range(1, n):
            if sets[t] != sets[t - 1]:
                times.update((t - 1, t))
        out += [f"  - name: {name}", "    interval: 1m", "    input_series:"] + input_block(ok, e500, e503)
        out.append("    alert_rule_test:")
        for t in sorted(times):
            for a, sev, _b, _l, _sh in genrules.ALERTS:
                out += [f"      - eval_time: {t}m", f"        alertname: {a}"]
                if fire[a][t]:
                    out += ["        exp_alerts:", "          - exp_labels:", f"              severity: {sev}",
                            "              job: api", "            exp_annotations:",
                            f"              summary: \"{genrules.SUMMARY[a]}\""]
                else:
                    out.append("        exp_alerts: []")
    return "\n".join(out) + "\n"


def cmd_suite():
    (HERE / "rules").mkdir(exist_ok=True)
    (HERE / "slo_test.yml").write_text(suite_text(), encoding="utf-8", newline="\n")
    rc, text = run_promtool(["test", "rules", "slo_test.yml"], HERE)
    print(text.strip()[:1500])
    print("exit code", rc)


def cmd_mutants():
    text = suite_text()
    base = HERE / "mut"
    shutil.rmtree(base, ignore_errors=True)

    def one(name):
        d = base / name
        (d / "rules").mkdir(parents=True)
        (d / "rules" / "slo.yml").write_text(genrules.slo_rules(None if name == "clean" else name),
                                             encoding="utf-8", newline="\n")
        (d / "slo_test.yml").write_text(text, encoding="utf-8", newline="\n")
        rc, out = run_promtool(["test", "rules", "slo_test.yml"], d)
        os.makedirs(HERE / "out", exist_ok=True)
        (HERE / "out" / f"mut_{name}.txt").write_text(out, encoding="utf-8")
        return name, rc, parse_failures(out)

    names = ["clean"] + genrules.MUTANTS
    with ThreadPoolExecutor(max_workers=6) as ex:
        results = list(ex.map(one, names))
    caught = 0
    for name, rc, fails in results:
        first = f"{fails[0][0]} at {fails[0][1]}m" if fails else "-"
        if name != "clean" and rc != 0:
            caught += 1
        print(f"{name:<15} exit {rc}  failed checks {len(fails):>3}  first: {first}")
    print(f"planted mistakes caught: {caught} of {len(genrules.MUTANTS)}; clean rules exit code {results[0][1]}")


if __name__ == "__main__":
    {"matrix": cmd_matrix, "suite": cmd_suite, "mutants": cmd_mutants}[sys.argv[1]]()

Step 6: Replay the traffic and read the results

Run the replay. All five scenarios run in parallel and take about ten seconds on a machine with 14 logical processors:

python lab.py matrix > matrix.txt
grep -E "^scenario|^ +\(min in\)|HighErrorRate|BurnFast1hOnly" matrix.txt

The columns are: first fire, the minute the alert first fired, counted from the first minute that contains a failing request (+0 is that very minute); budget used, the percentage of the 30-day budget consumed by the incident’s failing requests up to that minute (the healthy background failures are not counted); and minutes firing, how many one-minute evaluations the alert was firing during the replay.

What the usual alert does

scenario  alert           first fire budget used  minutes  model first fire
                            (min in)   (percent)   firing
outage15  HighErrorRate           +0         2.3       18  +0
outage15  BurnFast1hOnly          +0         2.3       73  +0
blip      HighErrorRate           +0         0.7        4  +0
blip      BurnFast1hOnly       never           -        0  never
burn5x    HighErrorRate           +0         0.0     2883  +0
burn5x    BurnFast1hOnly       never           -        0  never
burn7x    HighErrorRate           +0         0.0      723  +0
burn7x    BurnFast1hOnly       never           -        0  never
oneerror  HighErrorRate           +0         2.3        4  +0
oneerror  BurnFast1hOnly          +0         2.3       59  +0

The five-minute alert (HighErrorRate) fires at the first failing minute in every scenario. For the one-minute blip it pages for four minutes, although the blip used only 0.7 percent of the budget. For the slow leak it pages for 2,883 minutes, the whole two days plus the few minutes the window needs to clear, starting when 0.0 percent of the budget was gone. A person woken up by that page cannot tell from the page whether the budget is going to last six days or six minutes. The one-hour-only alert (BurnFast1hOnly) ignores the blip and the leaks, but it fires for 73 minutes in total for the 15-minute outage, because the hour-long window stays above the threshold long after the service has recovered.

What the burn-rate alerts do

grep -E "^scenario|^ +\(min in\)|SLOBurn|^model and" matrix.txt
scenario  alert           first fire budget used  minutes  model first fire
                            (min in)   (percent)   firing
outage15  SLOBurnFast             +0         2.3       18  +0
outage15  SLOBurnSlow             +2         6.9       41  +2
outage15  SLOBurnTicket           +3         9.3      370  +3
blip      SLOBurnFast          never           -        0  never
blip      SLOBurnSlow          never           -        0  never
blip      SLOBurnTicket        never           -        0  never
burn5x    SLOBurnFast          never           -        0  never
burn5x    SLOBurnSlow          never           -        0  never
burn5x    SLOBurnTicket         +719         8.3     2460  +719
burn7x    SLOBurnFast          never           -        0  never
burn7x    SLOBurnSlow           +299         4.9      425  +299
burn7x    SLOBurnTicket         +514         8.3      522  +514
oneerror  SLOBurnFast             +0         2.3        4  +0
oneerror  SLOBurnSlow          never           -        0  never
oneerror  SLOBurnTicket        never           -        0  never
model and promtool fire in exactly the same minutes for 25 of 25 scenario/alert pairs; largest first-fire difference 0 minute(s)

Go through the scenarios one by one:

  • outage15. All three alerts fire, one after another. SLOBurnFast fires in the first failing minute, when a single minute of total outage has already used 2.3 percent of the budget. SLOBurnSlow follows two minutes later at 6.9 percent, and the ticket three minutes in at 9.3 percent. The fast alert fires for 18 minutes, against 73 for the one-hour-only version: the five-minute window ends the alert about four minutes after the failures stop.
  • blip. No burn-rate alert fires at all. The usual alert paged for four minutes.
  • burn5x. No page. The ticket opens after 719 minutes, about 12 hours, when the incident has used 8.3 percent of the budget. The healthy background failures use roughly the other 1.7 percent, which brings the three-day ratio to the 0.1 percent threshold, the 10 percent row of the workbook table.
  • burn7x. A slightly faster leak does page, through the slow alert, after 299 minutes (about five hours) with 4.9 percent of the budget used, which is the 5 percent row of the table. The ticket follows at minute 514.
  • oneerror. At one request a minute, a single failed request pages. That is honest arithmetic and not a bug in the rule: at one request a minute the whole month’s budget is only about 43 failed requests, so one failure is 2.3 percent of it. Low-traffic services need a different approach, and the pitfalls section at the end shows the workbook’s options.

Step 7: Check promtool against an independent model

How do you know your expectations of an alert are right? One way is to calculate them a second way. model() in lab.py does the same job in plain Python. For each minute it computes the failure ratio over each window as the failing requests in the window divided by all requests in the window, applies the same thresholds and combines them with the same and.

There is one subtle detail. A window of five minutes does not contain five one-minute intervals. Since Prometheus 3.0 range selectors are left-open: the migration guide says “Lookback and range selectors are now left-open and right-closed (previously left-closed and right-closed)”. With samples exactly one minute apart, a [5m] range holds five samples: the guide says such queries “will always return 5 samples given even spacing”. Five samples are four intervals, so the model uses the window length minus one intervals.

The last line of the Step 6 output compares the two calculations over all five scenarios and all five alerts:

grep "^model and" matrix.txt

It reads: model and promtool fire in exactly the same minutes for 25 of 25 scenario/alert pairs, with a largest difference in first-fire time of 0 minutes. When two independent calculations agree minute for minute, you can trust both the model and the test file generated from it.

Step 8: Turn the replay into unit tests

A replay is an experiment. A unit test is a promise that keeps being checked. lab.py suite writes slo_test.yml, a standard promtool test file, using the model’s answers as the expectations. The beginning of that file looks like this:

rule_files:
  - rules/slo.yml
evaluation_interval: 1m
tests:
  - name: outage15
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="api",code="200"}'
        values: '0+5999x4319 25909681+0x14 25915680+5999x479'
      - series: 'http_requests_total{job="api",code="500"}'
        values: '0+1x4319 4319+0x14 4320+1x479'
      - series: 'http_requests_total{job="api",code="503"}'
        values: '0+0x4319 6000+6000x14 90000+0x479'
    alert_rule_test:
      - eval_time: 4319m
        alertname: SLOBurnFast

The three series lines are the counters for the outage15 scenario. Notice how short the notation makes them: 0+5999x4319 is three days of healthy traffic, 25909681+0x14 is the fifteen minutes during which healthy requests stop, and so on. Below the input, each alert_rule_test says “at minute t, the alert X should be firing with these labels and annotations”, or, with exp_alerts: [], “it should not be firing”. The labels you list are the rule’s own labels plus the labels of the series (job), and you leave alertname out. The suite checks every minute at which any alert changes state and the minute before it, plus the minutes around the start of each incident, so it pins down both when alerts fire and when they stay quiet.

Run the suite:

python lab.py suite
Processing test file: slo_test.yml
  SUCCESS
exit code 0

promtool test rules exits with 0 when every check passes and 1 when something fails, which is all a CI job needs to know. The suite has 102 checks in four scenarios and takes about 20 seconds.

Step 9: Break the rules on purpose

A test suite that has never failed might be a test suite that cannot fail. So now you will damage the rules nine ways, each a mistake a tired engineer could really make, and check that the suite fails every time. Run python lab.py mutants. It writes a broken copy of rules/slo.yml for each mistake, runs the same slo_test.yml against it and prints one line per mistake:

Name What was changed
factor_low The fast alert’s 14.4 typed as 1.44
budget_99 The budget written as 0.01, as for a 99 percent SLO
or_not_and The fast alert fires when the long window or the short window is high
short_eq_long The fast alert’s short window copy-pasted from the long one, so both are one hour
wrong_severity The fast alert labelled ticket instead of page
only_500 Failures counted as code="500", which misses 503
rate_5s The five-minute recording rule written with a range of five seconds
wrong_cmp The ticket comparison flipped from > to <
for_15m A for: 15m added to the fast alert to reduce noise
clean           exit 0  failed checks   0  first: -
factor_low      exit 1  failed checks   7  first: SLOBurnFast at 4320m
budget_99       exit 1  failed checks  28  first: SLOBurnFast at 4320m
or_not_and      exit 1  failed checks   7  first: SLOBurnFast at 4338m
short_eq_long   exit 1  failed checks   5  first: SLOBurnFast at 4338m
wrong_severity  exit 1  failed checks   9  first: SLOBurnFast at 4320m
only_500        exit 1  failed checks  30  first: SLOBurnFast at 4320m
rate_5s         exit 1  failed checks   9  first: SLOBurnFast at 4320m
wrong_cmp       exit 1  failed checks  18  first: SLOBurnTicket at 4319m
for_15m         exit 1  failed checks   8  first: SLOBurnFast at 4320m
planted mistakes caught: 9 of 9; clean rules exit code 0

The first row is the unchanged rule file, which must pass with exit code 0. Every damaged copy exits with 1, and the line names the first failed check. To see what a failure looks like, here is the beginning of the full promtool output for the for_15m copy:

Processing test file: slo_test.yml
  FAILED:
    name: outage15,
    alertname: SLOBurnFast, time: 3d, 
        exp:[
            0:
              Labels:{alertname="SLOBurnFast", job="api", severity="page"}
              Annotations:{summary="api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)"}
            ], 
        got:[]
    name: outage15,
    alertname: SLOBurnFast, time: 3d1m, 
        exp:[
            0:
              Labels:{alertname="SLOBurnFast", job="api", severity="page"}
              Annotations:{summary="api is burning its 30-day error budget at 14.4x or faster (1h and 5m windows)"}
            ], 

Read it from the top. name: outage15 is the test group, alertname: SLOBurnFast, time: 3d is the check that failed (3d is minute 4,320, the first failing minute of the outage), exp is what the test expected, and got:[] says the rule produced no alert. With for: 15m the alert stays pending for 15 minutes before it fires, so the check at the first failing minute finds nothing. It is exactly the kind of well-meant change that a green deploy would not reveal.

Common mistakes and how to avoid them

Putting recording rules and alerts in different groups

The documentation promises an order only inside a group: rules run one after another with the same evaluation time. Do not rely on any order between groups, because an alert in one group may then see a ratio that was computed at a different moment. Keep the recording rules above the alerts that use them in a single group, as rules/slo.yml does.

Adding for to a page alert without thinking

The Prometheus docs say “Alerting rules without the for clause will become active on the first evaluation.” The for_15m mistake above shows what a delay costs. Multi-window alerts already have a built-in delay, because the long window must exceed the threshold, so extra for time is usually unnecessary. If your alerts flap, look at keep_firing_for in the alerting rules documentation, which keeps an alert firing for a while after its condition stops being true.

A range window with too few samples

The rate_5s mistake writes [5s] against counters that are sampled once a minute. A five-second range holds at most one sample, and rate() cannot work out a rate from a single sample, so the recording rule produces no output and the alerts that depend on it never fire. Keep every window at least a few scrape intervals long, and let the unit test tell you when you do not.

Testing only the loud cases

Most people write a test for the outage and stop there. The blip scenario is where you learn that the alerts stay quiet, and an over-sensitive mistake such as factor_low can only be caught by a check that expects silence.

A budget typed for the wrong SLO

The budget_99 mistake looks plausible in a code review because 0.01 is a normal-looking number. Set the budget in one place and derive every threshold from it, as genrules.py does, and let slo_math.py show you the resulting percentages.

Low-traffic services

The oneerror scenario paged for one failed request. The workbook lists options for low-traffic services, the first of which is: “Generate artificial traffic to compensate for the lack of signal from real users.” The others are to combine smaller services for monitoring and to change the product so that one failure counts for less. Pick one before trusting a burn-rate alert on a service that handles a few requests an hour.

Silence when the metric disappears

These rules divide failures by all requests. If a service stops exporting its counters altogether, there is nothing to divide, and a query with no data returns no result, so none of the three alerts would fire. This lab did not test that case. Add a separate alert for missing metrics, for example one built on the absent() function, and test it the same way.

Confirm everything works end to end

From the slo-lab folder, these four checks should all pass:

  1. "$PROMTOOL" check rules rules/naive.yml rules/slo.yml reports SUCCESS for both files.
  2. python lab.py matrix | tail -1 says that the model and promtool fire in the same minutes for 25 of 25 pairs.
  3. python lab.py suite prints SUCCESS and exits with 0.
  4. python lab.py mutants ends with planted mistakes caught: 9 of 9.

To use this in a real repository, keep your rule files and a test file in version control and make CI run promtool check rules followed by promtool test rules. A non-zero exit code from either command should fail the build.

Where to go next

  • Route severity: page and severity: ticket in Alertmanager to different receivers, and decide whether a firing page should silence the matching ticket.
  • Replace the made-up scenarios with the real shape of your traffic, including its daily rhythm, and keep the incident scenarios you have lived through as permanent tests.
  • Apply the same burn-rate structure to a latency objective, by counting requests slower than your threshold as the “failures”.
  • If your failures come from slow or failing dependencies, our tutorials on HTTP timeouts and deadlines, retries with backoff and jitter and circuit breakers cover the fixes, and the alerts you built here tell you when they are needed.
  • For the metric side, see how to turn Prometheus histograms into OTLP metrics.

Sources

  • Google SRE Workbook, chapter 5, Alerting on SLOs (Table 5-8, the short-window rule of thumb and the low-traffic options).
  • Prometheus documentation: unit testing for rules, alerting rules, recording rules, query functions and the Prometheus 3.0 migration guide.
  • Prometheus 3.16.0 release on GitHub, the version used for every output in this post.

Tags:

AlertingError BudgetObservabilityPrometheuspromtoolReliability EngineeringSLO

Share

Three buckets on a wet wooden floor collecting water that leaked into a ship interior
Previous Post

WIRED’s “AI-Fueled Ransomware Boom” Rests on Leak-Site Counts That Reproduce at 25% to 39%, but the ReliaQuest Report Behind Its New-Group Numbers Calls the Rise Unlikely to Reflect a Real Increase in Activity and Never Mentions AI, and in a Public Tracker 97% of the Rise Comes From Groups First Seen in Q3

Amber flexible printed circuit connecting a display panel to a red rigid circuit board
Next Post

Trezor Confirms Cellular Implants in Three Safe 5 Wallets Sold by One Reseller, but Two Were Identified From Owners’ Photos, the 2.13.1 Changelog Never Mentions the Fix, and Recovery Words Captured Earlier Stay Exposed

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
16 Nov
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
16 Nov
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
Trending
November 16, 2026
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 15, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
November 15, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
November 15, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query

Related Posts

A laptop wrapped in a chain and padlock, illustrating least-privilege controls for AI agents.
Learning Hub

How to Secure Tool-Using AI Agents Before They Touch Production

June 8, 2026
Colorful sticky notes arranged on an office wall, symbolizing governance checklists and planning.
Learning Hub

AI Governance for Agentic Apps: A Practical Checklist for Builders

June 8, 2026
A technician connects green fiber optic cables at a data center, representing a private production inference endpoint.
Learning Hub

How to Deploy a Fine-Tuned LLM Behind a Private Production Inference Endpoint

June 8, 2026
Narrow aisle behind black supercomputer racks in a data center
Learning Hub

Kubernetes SELinux Volume Labeling: What Cluster Operators Should Audit Before v1.37

June 8, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026