How to Catch Vacuous AI-Generated Tests With Mutation Testing in Python

An AI coding agent fixes a bug, then writes a test for the fix. The test runs. It passes. You commit it, confident the bug cannot come back. Weeks later someone touches that same code path, the exact bug returns, and the test suite stays green the entire time. This tutorial teaches you how to catch that failure mode before it reaches production, using a technique called mutation testing: instead of asking whether your tests pass, you deliberately introduce a small, realistic bug into your code and ask whether your tests notice. If they do not notice, the tests were never really checking anything in the first place.

By the end of this tutorial you will have personally: written a function with one subtle, easy-to-get-wrong comparison, built a small mutation-testing tool from scratch using Python’s built-in ast module, used it to prove that a perfectly normal-looking test suite verifies nothing, asked a real local AI model to write its own test for that same function and mutation-tested that too, and fixed the AI’s test twice before it actually earned a passing score. Every command below was run for real while writing this post. The output shown is what actually printed, not a guess at what should print.

What “Vacuous” Means, and Why AI Agents Keep Writing These Tests

A vacuous test (also called a tautological test) is a test that passes no matter what the code underneath it actually does. The classic shape looks like assert result is not None or assert isinstance(result, list): it confirms the function ran without crashing, but says nothing about whether the answer is correct. AI coding agents produce these constantly, not because the underlying model is bad at testing, but because “write a test for this” is an underspecified request. Given a function and told to test it, the fastest way to produce something that passes is to assert the loosest possible thing about the output. A green checkmark then means “this code did not crash today,” which is a much weaker claim than “this code is correct,” and the two are easy to confuse from the outside.

Mutation testing is how you tell the difference. The idea, which predates AI coding agents by decades, is simple: take your source code, deliberately introduce one small, realistic bug called a mutant, and re-run your test suite against the mutated code. Two things can happen. If a test fails, the mutant is killed: your tests noticed the bug, which is what you want. If every test still passes, the mutant survived: your tests cannot tell the difference between correct code and this specific broken version of it, which means that logic is effectively untested no matter how green your test run looks. The share of mutants a suite kills is its mutation score, and it is a much harder number to fake than code coverage, since coverage only asks whether a line executed, not whether anything checked its result.

Prerequisites

• Python 3.9 or newer, since this tutorial relies on ast.unparse(), added to the standard library in Python 3.9. This tutorial was built and tested on Python 3.13.14.
• pip, to install pytest.
• Basic familiarity with writing a pytest test function: an assert statement inside a def test_...(): function. No prior experience with the ast module is assumed; this tutorial explains every part of it you will use.
• Optional, for the AI-agent portion: a local model running under Ollama. This tutorial uses qwen3.5:4b so every result is fully reproducible offline, but pasting the same function into ChatGPT, Claude, or Copilot and comparing notes works just as well.
• A terminal. Everything here works the same on Linux, macOS, and Windows (this tutorial was run on Windows, inside Git Bash).

One honest caveat up front. The most widely used production mutation-testing tool for Python, mutmut, states directly in its own documentation that it “must be run on a system with fork support,” and that “if you want to run on windows, you must run inside WSL.” This tutorial hit that exact wall while researching it (no WSL installed here), which is part of why the tool you will build below is a small, from-scratch mutation tester using nothing but the standard library. It teaches the same core idea mutmut and other production tools use, it runs natively on Windows, macOS, or Linux, and by the end you will understand mutation testing well enough to pick up mutmut, or its more heavyweight cousin cosmic-ray, without the internals feeling like a black box.

Step 1: Set Up the Project

mkdir mutation-demo
cd mutation-demo
pip install pytest

pip will print a longer installation log; the line that matters is the confirmation. Check it with:

python -m pytest --version

pytest 9.1.1

Any recent pytest 7.x, 8.x, or 9.x release works the same way for this tutorial.

Step 2: Write the Function You Will Test

Save the following as interval_merge.py. It takes a list of (start, end) tuples and merges any that overlap into the smallest equivalent set, a common building block behind calendar scheduling, log-range compaction, and rate-limit windows.

def merge_intervals(intervals):
    """Merge overlapping (start, end) intervals into the smallest equivalent set."""
    if not intervals:
        return []

    ordered = sorted(intervals, key=lambda iv: iv[0])
    merged = [ordered[0]]

    for start, end in ordered[1:]:
        last_start, last_end = merged[-1]
        if start <= last_end:
            merged[-1] = (last_start, max(last_end, end))
        else:
            merged.append((start, end))

    return merged

The function sorts the intervals by start time, then walks through them, extending the last merged interval whenever the next one overlaps it. The single line doing the real work is if start <= last_end:. That comparison decides whether two intervals count as overlapping, and it is exactly the kind of one-character decision that is easy to flip during a refactor without anyone, human or AI, noticing right away. That line is the target for everything that follows.

Step 3: Write a Test That Looks Reasonable, But Proves Nothing

Save this as test_naive.py. It is the kind of test a rushed developer, or an AI agent given a vague “add a test” instruction, tends to produce first.

from interval_merge import merge_intervals


def test_merge_intervals_runs():
    result = merge_intervals([(1, 3), (2, 6), (8, 10)])
    assert result is not None
    assert isinstance(result, list)
    assert len(result) <= 3

Run it against the correct implementation first, to confirm it passes before you do anything else:

python -m pytest test_naive.py -q

.                                                                        [100%]
1 passed in 0.02s

Why This Test Is Vacuous

Read the three assertions again. result is not None is true for almost any return value. isinstance(result, list) only checks the type, not the contents. len(result) <= 3 is a loose upper bound: the input has 3 intervals, so any correct or incorrect merging behavior that does not somehow produce more than 3 intervals will satisfy it. Not one assertion pins down what the merged intervals should actually look like. You will prove this empirically in Step 5, but you can already see the shape of the trap: this test will keep passing even if merge_intervals is rewritten to do something quietly wrong.

Step 4: Build a Minimal Mutation Tester From Scratch

Python’s ast module parses source code into an abstract syntax tree, a structure of Python objects that mirrors the code’s grammar: a comparison like start <= last_end becomes a Compare node holding the left side, the operator, and the right side. Because it is just a tree of objects, you can walk it, find the operator you want to change, swap it for a different one, and turn the modified tree back into runnable source code with ast.unparse(). That is the entire mechanism behind mutation testing: parse, swap one operator, unparse, run the tests, put the original back.

Finding What to Mutate

Start with a table of plausible operator swaps, and a small visitor class that walks the tree looking for comparisons using one of those operators. ast.NodeVisitor is a standard-library helper that calls a method named visit_<NodeType> for every node of that type it finds while walking the tree; overriding visit_Compare is enough to catch every comparison in the file.

import argparse
import ast
import copy
import shutil
import subprocess
import sys

TARGET = "interval_merge.py"
BACKUP = "interval_merge.py.orig"

# operator -> list of (mutant_name, replacement_node_type)
COMPARE_MUTATIONS = {
    ast.Lt: [("Lt -> LtE", ast.LtE), ("Lt -> Gt", ast.Gt)],
    ast.LtE: [("LtE -> Lt", ast.Lt), ("LtE -> GtE", ast.GtE)],
    ast.Gt: [("Gt -> GtE", ast.GtE), ("Gt -> Lt", ast.Lt)],
    ast.GtE: [("GtE -> Gt", ast.Gt), ("GtE -> LtE", ast.LtE)],
    ast.Eq: [("Eq -> NotEq", ast.NotEq)],
    ast.NotEq: [("NotEq -> Eq", ast.Eq)],
}


class CompareOpCollector(ast.NodeVisitor):
    def __init__(self):
        self.found = []  # list of (node, op_index, op_type)

    def visit_Compare(self, node):
        for i, op in enumerate(node.ops):
            if type(op) in COMPARE_MUTATIONS:
                self.found.append((node, i, type(op)))
        self.generic_visit(node)

Each comparison operator maps to one or two believable replacements rather than every mathematically possible one. Swapping <= for < or >= mirrors the kind of off-by-one mistake a person or a model actually makes; swapping it for something unrelated would not teach you anything about your tests. node.ops is a list because Python allows chained comparisons like a < b < c, so a single Compare node can hold more than one operator; this tutorial’s target function only ever has one, but the collector handles both cases correctly by indexing into that list.

Generating and Applying One Mutant at a Time

For every operator the collector finds, build a full copy of the tree with just that one operator swapped, then turn it back into source code. Running each mutant means writing it over the real file, running pytest against it in a subprocess, and reading the exit code: a return code of 0 means every test passed, meaning the mutant survived.

def build_mutants(source_path):
    with open(source_path, "r", encoding="utf-8") as f:
        source = f.read()
    tree = ast.parse(source)
    collector = CompareOpCollector()
    collector.visit(tree)

    mutants = []
    for node, op_index, op_type in collector.found:
        for label, replacement_cls in COMPARE_MUTATIONS[op_type]:
            mutant_tree = copy.deepcopy(tree)
            mutant_collector = CompareOpCollector()
            mutant_collector.visit(mutant_tree)
            target_node, target_index, _ = mutant_collector.found[
                collector.found.index((node, op_index, op_type))
            ]
            target_node.ops[target_index] = replacement_cls()
            ast.fix_missing_locations(mutant_tree)
            mutant_source = ast.unparse(mutant_tree)
            line_no = node.lineno
            mutants.append((f"line {line_no}: {label}", mutant_source))
    return source, mutants


def run_pytest(test_file):
    result = subprocess.run(
        [sys.executable, "-m", "pytest", test_file, "-q"],
        capture_output=True,
        text=True,
        timeout=30,
    )
    return result.returncode == 0

Two details matter here. First, copy.deepcopy(tree) runs once per mutant so that changing one operator never leaks into the next mutant; each one starts from a clean copy of the original tree. Second, ast.fix_missing_locations() fills in line and column numbers on the newly created operator node. Python’s own documentation explains why this matters: “when you compile a node tree with compile(), the compiler expects lineno and col_offset attributes for every node that supports them,” and this helper is the standard way to satisfy that expectation on nodes you build by hand instead of parsing from source. This script only unparses the tree rather than compiling it directly, but calling it is still the correct habit, since the moment you reuse this pattern anywhere that does compile a hand-edited tree, a missing location attribute becomes exactly the kind of error that is annoying to track down.

Tying It Together: the Main Loop

The last piece backs up the real source file, then for each mutant: writes the mutated source over the original, runs the target test file against it, records killed or survived, and restores the backup. The restore lives in a finally block on purpose, so that even if a mutant run crashes or the process is interrupted, the original correct file is never left overwritten on disk.

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("test_file")
    args = parser.parse_args()

    original_source, mutants = build_mutants(TARGET)
    shutil.copyfile(TARGET, BACKUP)

    print(f"Generated {len(mutants)} mutants from {TARGET}")
    print(f"Running each against {args.test_file}\n")

    killed = 0
    survived = []
    try:
        for label, mutant_source in mutants:
            with open(TARGET, "w", encoding="utf-8") as f:
                f.write(mutant_source)
            passed = run_pytest(args.test_file)
            if passed:
                survived.append(label)
                print(f"  SURVIVED  {label}  (tests still passed against buggy code)")
            else:
                killed += 1
                print(f"  killed    {label}")
    finally:
        shutil.copyfile(BACKUP, TARGET)

    total = len(mutants)
    score = (killed / total * 100) if total else 0.0
    print(f"\nMutation score: {killed}/{total} killed ({score:.1f}%)")
    if survived:
        print(f"{len(survived)} mutant(s) survived, meaning {args.test_file} never actually checks that logic:")
        for s in survived:
            print(f"  - {s}")


if __name__ == "__main__":
    main()

Save all of this as mutate.py in the same directory. This same backup-mutate-test-restore cycle is exactly how mutmut’s own documentation describes its workflow too, down to the warning to keep your real source file under version control before running it; you are not taking a shortcut here, you are looking at the same mechanism a production tool uses, minus its caching, parallelism, and much larger catalog of mutation types.

Step 5: Run the Mutation Tester Against the Naive Test

python mutate.py test_naive.py

Generated 2 mutants from interval_merge.py
Running each against test_naive.py

  SURVIVED  line 11: LtE -> Lt  (tests still passed against buggy code)
  SURVIVED  line 11: LtE -> GtE  (tests still passed against buggy code)

Mutation score: 0/2 killed (0.0%)
2 mutant(s) survived, meaning test_naive.py never actually checks that logic:
  - line 11: LtE -> Lt
  - line 11: LtE -> GtE

Zero percent. Both mutants of the one comparison in the whole file survived, and it is worth tracing through why, because the reason is more specific than “the assertions are weak.”

For the LtE -> Lt mutant (<= becomes <), run the same input, [(1, 3), (2, 6), (8, 10)], through the mutated logic by hand: comparing 2 against 3, 2 < 3 is still true, so the merge still happens exactly as before, and the mutated function returns the exact same result as the correct one for this specific input. No assertion, however strict, could catch this mutant without a different test case, because the two versions of the code are not actually different for this input.

For the LtE -> GtE mutant (<= becomes >=), the logic is inverted: intervals now “merge” only when they do not overlap. Tracing it by hand, the mutated function returns [(1, 3), (2, 10)], two intervals that visibly overlap each other, which defeats the entire purpose of the function. But that result still has length 2, so len(result) <= 3 never notices, and neither does isinstance or is not None. The test was never structurally capable of catching a bug that changes the values without changing the shape of the output.

Step 6: Write a Test Suite That Actually Verifies Behavior

Replace loose type and length checks with exact expected values across a handful of representative cases, including the empty list, a single interval, and the specific edge case where one interval’s end touches the next one’s start:

from interval_merge import merge_intervals


def test_no_intervals():
    assert merge_intervals([]) == []


def test_single_interval():
    assert merge_intervals([(1, 3)]) == [(1, 3)]


def test_overlapping_intervals_merge():
    assert merge_intervals([(1, 3), (2, 6)]) == [(1, 6)]


def test_touching_intervals_merge():
    assert merge_intervals([(1, 3), (3, 5)]) == [(1, 5)]


def test_disjoint_intervals_stay_separate():
    assert merge_intervals([(1, 2), (5, 6)]) == [(1, 2), (5, 6)]


def test_unsorted_input_is_sorted_before_merging():
    assert merge_intervals([(5, 6), (1, 2)]) == [(1, 2), (5, 6)]


def test_nested_interval_is_absorbed():
    assert merge_intervals([(1, 10), (2, 3)]) == [(1, 10)]

Save this as test_thorough.py, confirm it passes on the correct code, then mutation-test it:

python -m pytest test_thorough.py -q

.......                                                                  [100%]
7 passed in 0.02s

python mutate.py test_thorough.py

Generated 2 mutants from interval_merge.py
Running each against test_thorough.py

  killed    line 11: LtE -> Lt
  killed    line 11: LtE -> GtE

Mutation score: 2/2 killed (100.0%)

Both mutants are killed now. The decisive addition is test_touching_intervals_merge: (1, 3) and (3, 5) touch exactly at the boundary the <= comparison exists to handle, so a mutant that changes <= to < produces a genuinely different, wrong result on that specific input, [(1, 3), (3, 5)] instead of the correct [(1, 5)]. Exact-value assertions mattered, but so did choosing an input that actually exercises the line being tested; you will see that same lesson resurface in Step 8.

Step 7: Ask a Real AI Agent to Write Its Own Test

To see whether this is a real phenomenon and not just a hypothetical, this tutorial sent the same function to a local qwen3.5:4b model running under Ollama and asked it to write a test, the same way a developer might casually delegate a “quick test” to an AI coding assistant mid-task.

The First Attempt Did Not Even Run

The first prompt simply asked the model to “write a quick pytest test for this function to make sure it works.” Here is the real, complete response, unedited:

import pytest
from typing import List, Tuple


def merge_intervals(intervals: List[Tuple[int, int]]) -> List[Tuple[int, int]]:
    if not intervals:
        return []
    ordered = sorted(intervals, key=lambda iv: iv[0])
    merged = [ordered[0]]
    for start, end in ordered[1:]:
        last_start, last_end = merged[-1]
        if start <= last_end:
            merged[-1] = (last_start, max(last_end, end))
        else:
            merged.append((start, end))
    return merged


def test_empty_intervals():
    assert merge_intervals([]) == []


def test_single_interval():
    result = merge_intervals([(1, 5)])
    assert result == [(1, 5)]


def test_disjoint_intervals():
    intervals = [(1, 2), (3, 4), (5, 6)]
    expected = sorted(intervals)
    assert merge_intervals(intervals) == expected


def test_overlapping_intervals():
    result = merge_intervals([(1, 3), (2, 6)])
    assert result == [(1, 6)]


def test_nested_intervals():
    intervals = [(0, 5), (4, 7)]
    expected = sorted(intervals)
    result = merge_intervals(intervals)
    for m in result:
        if isinstance(m[1], int):
            assert len(result[m]) > 1


def test_complex_overlapping():
    intervals = [(0, 2), (3, 4], 

This file cannot even be collected by pytest, for two separate reasons worth naming individually. The last test is a plain syntax error: a mismatched bracket and a trailing comma cut the response off mid-statement. The second-to-last test, test_nested_intervals, is syntactically valid Python that would still crash at runtime: m is a tuple like (1, 10), and result[m] tries to use that tuple as a list index, which raises TypeError: list indices must be integers or slices, not tuple. Before you can ask whether an AI-generated test verifies anything, you sometimes have to get past the more basic question of whether it runs at all.

The Second Attempt Ran and Passed, and Still Verified Nothing

A more constrained prompt, asking for exactly one short “smoke test” confirming the function does not crash, produced clean, valid Python:

import pytest
from interval_merge import merge_intervals


def test_merge_intervals_smoke():
    intervals = [(1, 3), (2, 6), (8, 10)]
    result = merge_intervals(intervals)
    assert isinstance(result, list)
    assert len(result) == 2

Save this as test_ai_smoke.py. It looks more specific than the earlier naive test, since it pins down an exact expected count instead of a loose upper bound. Confirm it passes, then mutation-test it anyway:

python -m pytest test_ai_smoke.py -q

.                                                                        [100%]
1 passed in 0.02s

python mutate.py test_ai_smoke.py

Generated 2 mutants from interval_merge.py
Running each against test_ai_smoke.py

  SURVIVED  line 11: LtE -> Lt  (tests still passed against buggy code)
  SURVIVED  line 11: LtE -> GtE  (tests still passed against buggy code)

Mutation score: 0/2 killed (0.0%)
2 mutant(s) survived, meaning test_ai_smoke.py never actually checks that logic:
  - line 11: LtE -> Lt
  - line 11: LtE -> GtE

Zero percent again, for the same underlying reasons as Step 5. The LtE -> Lt mutant produces an identical result for this exact input, so no assertion on this input could catch it. The LtE -> GtE mutant produces [(1, 3), (2, 10)], wrong and overlapping, but still exactly 2 items, so len(result) == 2 passes right over it. An assertion that looks like it checks something specific still checked the wrong thing, because the input never forced the boundary condition to matter.

Step 8: Fix the AI’s Test, Twice

The obvious first fix is to assert the exact expected values instead of just the count:

from interval_merge import merge_intervals


def test_merge_intervals_smoke():
    intervals = [(1, 3), (2, 6), (8, 10)]
    result = merge_intervals(intervals)
    assert result == [(1, 6), (8, 10)]

python mutate.py test_ai_smoke.py

Generated 2 mutants from interval_merge.py
Running each against test_ai_smoke.py

  SURVIVED  line 11: LtE -> Lt  (tests still passed against buggy code)
  killed    line 11: LtE -> GtE

Mutation score: 1/2 killed (50.0%)
1 mutant(s) survived, meaning test_ai_smoke.py never actually checks that logic:
  - line 11: LtE -> Lt

Better, but not fixed. The GtE mutant is caught now, since [(1, 3), (2, 10)] no longer equals the exact expected list. But the Lt mutant still survives, for the exact reason traced through in Step 5: on this specific input, <= and < produce identical output. Tightening the assertion did not help, because the input was never the kind of input that could distinguish the two operators in the first place. Add a second test using the same touching-boundary input from Step 6:

from interval_merge import merge_intervals


def test_merge_intervals_smoke():
    intervals = [(1, 3), (2, 6), (8, 10)]
    result = merge_intervals(intervals)
    assert result == [(1, 6), (8, 10)]


def test_merge_intervals_touching_boundary():
    intervals = [(1, 3), (3, 5)]
    result = merge_intervals(intervals)
    assert result == [(1, 5)]

python -m pytest test_ai_smoke.py -q

..                                                                       [100%]
2 passed in 0.02s

python mutate.py test_ai_smoke.py

Generated 2 mutants from interval_merge.py
Running each against test_ai_smoke.py

  killed    line 11: LtE -> Lt
  killed    line 11: LtE -> GtE

Mutation score: 2/2 killed (100.0%)

100 percent. Getting there took two separate corrections: first replacing a shape check with a value check, then replacing an input that avoided the boundary with one that hits it directly. Both were necessary; neither alone was enough.

Common Mistakes and Gotchas

• Checking type, length, or non-None instead of exact values. This is the fastest way to write a test that always passes, as Steps 3, 5, and the first half of Step 8 all showed directly.
• Checking exact values on an input that never exercises the edge case your logic branches on. This is the subtler trap from Step 8: a strict-looking assertion is still vacuous with respect to a specific bug if the input can never trigger that bug in the first place.
• Trusting AI-generated tests without running them at all. The first agent response in Step 7 would not even pass pytest’s collection phase. Reading generated code and assuming it works is not the same as executing it.
• Skipping the restore-on-failure step when writing your own mutation tooling. The finally block in main() exists because a crashed or interrupted run should never leave your real source file overwritten with a mutant. Verify this yourself by checking interval_merge.py still contains <=, not a mutated operator, after every run in this tutorial.
• Expecting this to scale to a large codebase as-is. This tool reruns the entire target test file once per mutant, sequentially, with no caching. That is fine for one function and two mutants; a file with dozens of comparisons and a slow test suite would take a long time. Mutmut’s own documentation lists remembering prior results and parallel execution as core features for exactly this reason.

Verify Everything End to End

Confirm the whole chain still works from a clean state by rerunning both final commands back to back:

python -m pytest test_thorough.py test_ai_smoke.py -q

.........                                                                [100%]
9 passed in 0.02s

python mutate.py test_thorough.py && python mutate.py test_ai_smoke.py

Both runs should print Mutation score: 2/2 killed (100.0%), and interval_merge.py should still contain the original if start <= last_end: line when you check it afterward, confirming the backup-and-restore cycle left nothing behind.

Next Steps

• If you have access to Linux, macOS, or WSL on Windows, install the real mutmut (pip install mutmut, then mutmut run) and compare its report against your own tool on the same function.
• Extend COMPARE_MUTATIONS to cover arithmetic operators (ast.Add, ast.Sub) and boolean literals; mutmut’s own documentation lists a much larger catalog of mutation types worth studying once this one clicks.
• Add a minimum mutation score check to your CI pipeline as a merge gate, the same way you might already gate on code coverage, but harder to satisfy with a hollow test.
• Point this tool at a test file an AI coding agent wrote for one of your own functions before you trust it. This tutorial found a real 0 percent score on the first honest attempt; yours might too.