How to Review AI-Generated Python Code Before You Merge It

A coding agent can write a working-looking Python module in under a minute. Reading it carefully enough to trust it takes a lot longer than that, and the gap between those two speeds is where bugs slip into production. This tutorial teaches a repeatable, five-step workflow for reviewing AI-generated Python code: understand what the code should do, run automated checks, read the code with a plan instead of top to bottom, hunt through the specific categories of mistakes coding agents make, then fix and verify.

To make this concrete, everything in this post runs against one real, unedited piece of code generated by a local model, standing in for a fast coding agent. Nothing here is staged. The bugs you’ll read about are the actual output of a real prompt, and every claim about what breaks (and why) is backed by a command you can rerun yourself and see fail or pass exactly as described.

What Changes When a Coding Agent Writes the Code

A code review is the process of looking at a change before you accept it, to confirm it does what it should, and that it’s correct, secure, and maintainable. None of that changes when the author is an agent instead of a teammate. What changes is the volume and the speed: an agent can hand you two hundred lines in the time it takes a human to write twenty, and most people start losing focus somewhere around four hundred lines of code or an hour of reading, whichever comes first. You become the bottleneck, so you need a plan for where to spend your limited attention instead of reading every line with equal care.

The kinds of mistakes shift too. A teammate’s bug usually looks like a bug: a missing case, a typo, an obviously incomplete function. AI-generated code tends to look finished. The logic can read as correct while doing the wrong thing. It can call a function or import a package that sounds plausible and doesn’t exist. It can wrap risky code in what looks like proper error handling while the try/except block sits in the wrong place and catches nothing. This tutorial’s five-step structure, and the specific bug categories in Step 5, follow the workflow described in Real Python’s How to Review AI-Generated Python Code Efficiently; everything you’ll actually run and verify below is original, built from scratch against a real generated sample.

Prerequisites

• Python 3.10 or newer. This tutorial was built and tested on Python 3.13.14.
• Comfortable reading Python: you don’t need to be an expert, but you should be able to follow a function line by line.
• pip to install the review tools: ruff, mypy, pytest, and bandit. Exact versions used here: ruff 0.16.8, mypy 2.3.1, pytest 9.1.1, bandit 1.9.4.
• Optional: Ollama installed locally with the qwen2.5:1.5b model pulled, if you want to reproduce the code-generation step yourself instead of using the file provided below. You don’t need this to follow the review workflow; the generated file is included in full.

Start by installing the review tools in a fresh virtual environment:

python -m venv venv
venv\Scripts\activate
pip install ruff mypy pytest bandit

The Workflow You Will Follow

Five steps, in order:

1. Get the code and write down what it should do before you read a single line of the implementation.
2. Run automated checks first so a linter, a type checker, and a security scanner catch the mechanical problems before you spend human attention on them.
3. Read with a plan: start at the riskiest part of the code (I/O, error handling, the actual logic), not at line one.
4. Hunt through specific categories of mistakes coding agents commonly make, one category at a time, instead of reading once and hoping you noticed everything.
5. Fix what you found, then prove the fix with tests that fail against the original code and pass against the corrected version.

Step 1: Get a Real Piece of AI-Generated Code to Review

You’ll review a small, self-contained task: a function that scans a directory of log files, counts how many lines fall into each severity level, and writes a JSON summary. This is a realistic size for a single PR from a coding agent, complex enough to have real edge cases (malformed lines, an ignore-file convention, a missing directory) without being so large that the review itself becomes the tutorial.

How the code was generated

The exact task specification below was sent to a local qwen2.5:1.5b model (1.5B parameters, Q4_K_M quantization, Apache 2.0 licensed, from Alibaba Cloud) running under Ollama, with no temperature override, so the model’s default sampling settings apply:

Write a Python module named usage_report.py.

It must define one function:

    def generate_usage_report(log_dir: str, output_path: str) -> dict

Behavior:
- Scan log_dir for all files ending in ".log".
- Each line in a log file has the format: "TIMESTAMP LEVEL MESSAGE", for example:
  2026-09-16T10:00:00 ERROR Disk full on /var
  LEVEL is always one of: DEBUG, INFO, WARNING, ERROR, CRITICAL.
- If a file named ".reportignore" exists directly inside log_dir, treat each line in it
  as a filename (relative to log_dir) that should be skipped entirely.
- Count how many lines match each LEVEL, across every included file.
- Some lines will not match the expected "TIMESTAMP LEVEL MESSAGE" format. Skip those
  lines instead of crashing, but keep a running count of how many lines were skipped.
- After scanning everything, write a JSON summary to output_path with this exact shape:
    {
      "counts": {"DEBUG": 0, "INFO": 0, "WARNING": 0, "ERROR": 0, "CRITICAL": 0},
      "total_lines": 0,
      "skipped_lines": 0,
      "files_processed": 0
    }
- Return that same summary dict from the function.
- If log_dir does not exist, raise a clear, specific error.

Write production-quality Python code that a teammate could drop straight into a real
project. Only output the code for usage_report.py, no explanation before or after it.

This is exactly what came back, character for character, with the markdown fence stripped:

import os
import json
from typing import Dict

def generate_usage_report(log_dir: str, output_path: str) -> dict:
    def filter_skipped_lines(lines):
        skip = False
        for line in lines:
            if not (skip := ".reportignore" in line):
                yield line.strip()

    try:
        with open(os.path.join(output_path), "w") as outfile:
            counts = {"DEBUG": 0, "INFO": 0, "WARNING": 0, "ERROR": 0, "CRITICAL": 0}
            total_lines_skipped = 0

            for root, _, files in os.walk(log_dir):
                if ".reportignore" in os.listdir(root):
                    continue
                for file_name in filter_skipped_lines(files):
                    log_file_path = os.path.join(root, file_name)
                    with open(log_file_path, "r") as infile:
                        skip_lines = 0

                        while True:
                            line = next(infile, None)
                            if line is None:
                                break
                            timestamp, level, message = line.split(" ", 2)

                            try:
                                if ":" in level or "[" in level:
                                    counts[level] += 1
                                else:
                                    total_lines_skipped += 1

                            except ValueError as e:
                                skip_lines += 1
                                continue

                        outfile.write(f"{json.dumps(counts)}\n")
                        outfile.flush()

            return {"counts": counts, "total_lines": len(os.listdir(log_dir)), "skipped_lines": total_lines_skipped, "files_processed": len(files)}

    except FileNotFoundError as e:
        raise FileNotFoundError("Log directory not found.") from e

At a glance, this looks reasonable. It has a try/except around the risky parts, it has a helper function for filtering, it returns the right shape of dictionary. That’s exactly the trap: nothing here screams “broken” on a skim. Save this file as usage_report.py and let’s start reviewing it the way the workflow above describes, not by reading it top to bottom.

Step 2: Write Down What the Code Is Supposed to Do

Before opening the implementation, turn the task spec into a short checklist you can hold the code against. Do this from the spec, not from the code, so you’re comparing behavior to requirements instead of confirming whatever the code happens to do:

• Only files ending in .log should be scanned.
• A .reportignore file should exclude the specific filenames listed inside it, not everything in the directory.
• A malformed line should be skipped and counted, not crash the run.
• The returned dict and the JSON written to disk should have identical contents.
• A missing log_dir should raise a clear error.

Five checkable claims. Keep this list open while you read; every one of them turns out to matter.

Step 3: Run the Automated Checks First

Before reading a line, run the checks that a computer does better and faster than you do: style and import hygiene, type safety, and known-insecure patterns. Doing this first means that by the time you start reading, your attention is free for the things a linter can’t see.

Ruff: style, imports, and obvious mistakes

$ ruff check usage_report.py

I001 [*] Import block is un-sorted or un-formatted
UP035 `typing.Dict` is deprecated, use `dict` instead
F401 [*] `typing.Dict` imported but unused
F841 Local variable `skip` is assigned to but never used
RUF059 Unpacked variable `timestamp` is never used
RUF059 Unpacked variable `message` is never used
F841 [*] Local variable `e` is assigned to but never used

Found 7 errors.
[*] 3 fixable with the `--fix` option (2 hidden fixes can be enabled with the `--unsafe-fixes` option).

Two of these are worth pausing on because they’re early warning signs, not just style nits. Ruff’s own unused-variable (F841) rule flags skip and e: “a variable that is defined but not used is likely a mistake.” The skip variable comes from a walrus assignment (skip := ".reportignore" in line) that’s never read afterward, meaning that check is doing something other than what its name implies. Ruff’s unused-unpacked-variable (RUF059) rule flags timestamp and message: they’re unpacked from every line and then never referenced again, which is a strong hint that whatever validation was supposed to happen on them didn’t.

mypy --strict: type safety

$ mypy usage_report.py --strict

usage_report.py:5: error: Missing type arguments for generic type "dict"  [type-arg]
usage_report.py:6: error: Function is missing a type annotation  [no-untyped-def]
usage_report.py:20: error: Call to untyped function "filter_skipped_lines" in typed context  [no-untyped-call]
Found 3 errors in 1 file (checked 1 source file)

The mypy strict mode documentation describes --strict as enabling a defined bundle of checks, specifically --disallow-any-generics, --disallow-subclassing-any, --disallow-untyped-calls, --disallow-untyped-defs, --disallow-incomplete-defs, --check-untyped-defs, --disallow-untyped-decorators, --warn-redundant-casts, --warn-unused-ignores, --warn-return-any, --no-implicit-reexport, --strict-equality, and --extra-checks. None of what it caught here is a functional bug; it’s telling you the inner helper function has no type signature at all, which matters for readability but won’t crash anything.

Bandit: security patterns

$ bandit usage_report.py

Test results:
        No issues identified.

Code scanned:
        Total lines of code: 38
        Total lines skipped (#nosec): 0

Run metrics:
        Total issues (by severity):
                Undefined: 0
                Low: 0
                Medium: 0
                High: 0

Bandit’s own documentation describes it plainly: “Bandit is a tool designed to find common security issues in Python code. To do this, Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes.” There genuinely is nothing insecure here (no eval, no shell calls, no unsafe deserialization), so a clean Bandit run is the correct, honest result, not a sign the tool failed to find something.

What the automated checks missed

Ten findings total across three tools, and every single one of them is a style, typing, or naming issue. Not one of them caught that this function is going to return wrong data or crash on ordinary input. That’s not a weakness in the tools; it’s exactly the boundary the source article’s own framing draws: automated checks are for the mechanical stuff, and reading with a plan is for everything else. Keep your five-item checklist from Step 2 open, because none of it has been touched yet.

Step 4: Read With a Plan, Not Top to Bottom

Instead of reading from import os downward, start at the parts most likely to be wrong: the I/O boundaries (what gets read, what gets written), the error handling (does the except block actually wrap the thing that can fail?), and the logic that maps directly onto your checklist (the .reportignore handling, the level counting). Trace what happens for one concrete file with one concrete line, by hand, before running anything. If you do that here, two things should jump out even before you execute a single scenario:

• The try/except ValueError block starts after the line timestamp, level, message = line.split(" ", 2). If that specific line is what raises the ValueError (and it is, on any line without at least two spaces), the except block never gets a chance to catch it.
• The condition guarding counts[level] += 1 is if ":" in level or "[" in level. None of DEBUG, INFO, WARNING, ERROR, or CRITICAL contain a colon or a bracket. Follow that logic for a completely normal line and it takes the else branch every time.

Both of those are things you can find by tracing the code by hand. The next step confirms them by actually running it, because a hand-trace can still be wrong, and confirming by running is part of the workflow, not an optional extra.

Step 5: Hunt for the Bugs by Category

Reading once and hoping you noticed everything is how real bugs slip through. Going category by category, and actually running the code against each one, is what this step is for. Every result below is real, unedited output.

Architecture, Modularization, and Fit

The spec says “scan log_dir for all files ending in .log.” Reading the code, there is no .endswith(".log") check anywhere. Every file os.walk reports gets treated as a log file, and Python’s own os.walk documentation confirms why that’s dangerous: “filenames is a list of the names of the non-directory files in dirpath”, with no filtering of any kind, hidden files included.

Here’s what that produces with a directory containing one real log file and one unrelated text file:

>>> generate_usage_report("repro_ext", "repro_ext_out.json")
{'counts': {'DEBUG': 0, 'INFO': 0, 'WARNING': 0, 'ERROR': 0, 'CRITICAL': 0},
 'total_lines': 2, 'skipped_lines': 2, 'files_processed': 2}

files_processed is 2, even though only one of those two files was ever a log file. Worse: if you write the output report into the same directory you’re scanning (a completely natural thing to do), the tool creates its own output file before the scan even starts, then picks that half-written file back up as if it were a third input, because nothing filters by extension:

>>> generate_usage_report("repro_selfread", "repro_selfread/out.json")
{'counts': {...all zero...}, 'total_lines': 2, 'skipped_lines': 2, 'files_processed': 2}
>>> os.listdir("repro_selfread")
['app.log', 'out.json']

The tool read its own output back in as an input on the very first run. This is exactly the kind of “missing requirement” bug that a fit-and-scope read catches and an automated linter has no way to see.

Logic Bugs

Two separate logic bugs live in the .reportignore handling and the level counting. First, set up a directory the way the spec describes: app.log with real log lines, secrets.log that should be excluded, and a .reportignore file listing secrets.log:

>>> result = generate_usage_report("fixtures/sample_logs", "scenario1_output.json")
>>> result
{'counts': {'DEBUG': 0, 'INFO': 0, 'WARNING': 0, 'ERROR': 0, 'CRITICAL': 0},
 'total_lines': 3, 'skipped_lines': 0, 'files_processed': 3}
>>> os.path.getsize("scenario1_output.json")
0

No exception. A dict comes back that looks plausible. And the output file exists but is completely empty. Reading the code explains why: if ".reportignore" in os.listdir(root): continue checks whether a .reportignore file exists in the directory, and if it does, skips every file in that directory, full stop. It never opens .reportignore to read which specific filenames it names. The one time the spec’s ignore-file feature is actually exercised, the function silently processes nothing.

Second, take a directory with only well-formed lines, no malformed input at all:

>>> result = generate_usage_report("fixtures/sample_logs_clean", "scenario3_output.json")
>>> result
{'counts': {'DEBUG': 0, 'INFO': 0, 'WARNING': 0, 'ERROR': 0, 'CRITICAL': 0},
 'total_lines': 1, 'skipped_lines': 5, 'files_processed': 1}

Five valid lines went in. All five were counted as skipped, and every real count stayed at zero. This is the ":" in level or "[" in level condition from Step 4 confirmed: since no real severity level contains a colon or a bracket, that check is always false for correctly formatted input, so the code always takes the branch meant for bad data.

Missing Edge Cases and Swallowed Errors

The spec explicitly requires malformed lines to be skipped, not to crash the run. Add one deliberately truncated line (“CORRUPTED LINE”, which has only one space and can’t unpack into three parts) to an otherwise normal log file:

>>> generate_usage_report("fixtures/sample_logs_no_ignore", "scenario2_output.json")
Traceback (most recent call last):
  File "usage_report.py", line 29, in generate_usage_report
    timestamp, level, message = line.split(" ", 2)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 3, got 2)

This is the sharpest example in the whole file of code that looks like it handles the problem it doesn’t handle. There’s a try: ... except ValueError as e: block sitting right there, a few lines below the line that actually raises the ValueError. It looks like defensive error handling on a skim. It catches nothing, because the operation that can fail happens before the try block starts.

Security Weaknesses

Bandit already reported zero issues in Step 3, and a manual read confirms it: no eval, no string-built shell commands, no unsafe deserialization, no path built from unsanitized user input in a way that escapes log_dir. This category comes back clean, and it’s worth saying so explicitly rather than skipping it, because a category that never gets checked is a category you can’t actually vouch for.

Made-Up APIs and Packages

Also clean here: every call in this file is a real standard-library function (os.walk, os.listdir, json.dumps), nothing was imported that doesn’t exist. It’s worth checking this deliberately rather than assuming, because it’s a documented, measured failure mode for coding agents generally, not a hypothetical one. The Cloud Security Alliance’s research note on “slopsquatting” cites the original USENIX Security 2025 study, which generated 2.23 million code samples across 16 code-generating models and found that “440,445 (19.7%) contained at least one hallucinated package name,” with open-source models hallucinating at an average rate of 21.7 percent versus 5.2 percent for commercial models. The same research found that 43 percent of hallucinated package names reappeared on every single rerun of an identical prompt, meaning an attacker can predict and pre-register the names a model is likely to invent. The fix for this category isn’t a linter rule; it’s confirming with pip show <package> or an actual import that anything unfamiliar in a diff is real before you install it.

Performance and Resource Traps

The output file is opened for writing before the scan even begins, and stays open for the entire run. Combine that with the missing extension filter from the Architecture section, and you get the self-read behavior demonstrated above: the tool’s own in-progress output can become an unintended input. Separately, outfile.write() fires once per file processed instead of once at the end, so the file on disk accumulates multiple JSON objects rather than the single summary the spec calls for; you can see this directly by inspecting a run against two files:

>>> with open("repro_ext_out.json") as f:
...     print(repr(f.read()))
'{"DEBUG": 0, "INFO": 0, "WARNING": 0, "ERROR": 0, "CRITICAL": 0}\n{"DEBUG": 0, "INFO": 0, "WARNING": 0, "ERROR": 0, "CRITICAL": 0}\n'

Two lines of JSON is not one JSON document. Anything downstream that tries json.load() on this file breaks, which the next section demonstrates directly.

Concurrency and Async Bugs

Not applicable to this sample; nothing here is threaded, async, or otherwise concurrent. This category matters a lot when a coding agent adds concurrency to speed something up (a very common request), and it’s worth naming explicitly as “not applicable” rather than silently skipping it, the same way Security and Made-Up APIs came back clean above.

Maintainability Problems

Beyond what Ruff already caught in Step 3 (the unused import, the unused walrus variable, the two unused unpacked variables, the unused exception variable), two problems are visible only by reading: the helper function is named filter_skipped_lines but it’s applied to filenames, not lines, and it doesn’t actually filter based on the ignore file’s contents at all, just a substring check against the filename itself. And the final return statement computes "files_processed": len(files), where files is a loop variable from for root, _, files in os.walk(...) that still holds whatever value it had on the last directory os.walk visited, not a running count. A misleading name and a stale loop variable are exactly the kind of thing that reads fine at a glance and misleads the next person who touches this file.

Step 6: Fix the Issues, Then Verify With Tests

The corrected module

Here’s a version that addresses every finding above: a real .log extension filter, a .reportignore file that’s actually read and parsed into a set of filenames, level validation against the known set instead of a punctuation check, the split-and-unpack wrapped in the try block that’s supposed to catch it, a single JSON write at the end instead of one per file, and counters computed directly instead of derived from stale directory listings:

"""Summarize per-level line counts across a directory of plain-text log files."""

from __future__ import annotations

import json
import os

VALID_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")


def _load_ignore_set(log_dir: str) -> set[str]:
    """Read .reportignore (if present) into a set of filenames to skip."""
    ignore_path = os.path.join(log_dir, ".reportignore")
    if not os.path.isfile(ignore_path):
        return set()
    with open(ignore_path, "r", encoding="utf-8") as handle:
        return {line.strip() for line in handle if line.strip()}


def generate_usage_report(log_dir: str, output_path: str) -> dict[str, object]:
    """Count log lines per severity level and write a JSON summary.

    Raises FileNotFoundError if log_dir does not exist.
    """
    if not os.path.isdir(log_dir):
        raise FileNotFoundError(f"Log directory not found: {log_dir}")

    ignored_files = _load_ignore_set(log_dir)
    counts = {level: 0 for level in VALID_LEVELS}
    total_lines = 0
    skipped_lines = 0
    files_processed = 0

    for entry in sorted(os.listdir(log_dir)):
        if not entry.endswith(".log"):
            continue
        if entry in ignored_files:
            continue

        file_path = os.path.join(log_dir, entry)
        if not os.path.isfile(file_path):
            continue

        files_processed += 1
        with open(file_path, "r", encoding="utf-8") as handle:
            for raw_line in handle:
                line = raw_line.rstrip("\n")
                if not line:
                    continue
                total_lines += 1

                try:
                    _timestamp, level, _message = line.split(" ", 2)
                except ValueError:
                    skipped_lines += 1
                    continue

                if level not in counts:
                    skipped_lines += 1
                    continue

                counts[level] += 1

    summary: dict[str, object] = {
        "counts": counts,
        "total_lines": total_lines,
        "skipped_lines": skipped_lines,
        "files_processed": files_processed,
    }

    with open(output_path, "w", encoding="utf-8") as handle:
        json.dump(summary, handle, indent=2)

    return summary

Rerunning the same three scenarios from Step 5 against this version:

Scenario 1 (secrets.log excluded via .reportignore):
{'counts': {'DEBUG': 0, 'INFO': 1, 'WARNING': 1, 'ERROR': 1, 'CRITICAL': 1},
 'total_lines': 5, 'skipped_lines': 1, 'files_processed': 1}

Scenario 2 (malformed line, no ignore file):
{'counts': {'DEBUG': 0, 'INFO': 1, 'WARNING': 1, 'ERROR': 1, 'CRITICAL': 1},
 'total_lines': 5, 'skipped_lines': 1, 'files_processed': 1}
(no exception raised)

Scenario 3 (all well-formed lines, no ignore file):
{'counts': {'DEBUG': 0, 'INFO': 2, 'WARNING': 1, 'ERROR': 1, 'CRITICAL': 1},
 'total_lines': 5, 'skipped_lines': 0, 'files_processed': 1}

Scenario 4 (missing log_dir):
FileNotFoundError: Log directory not found: fixtures/does_not_exist

Every number matches what the checklist from Step 2 describes. The malformed line in Scenario 2 is skipped and counted instead of crashing the process. Secrets.log stays excluded in Scenario 1, and this time the exclusion is because its filename is actually in the ignore set, not because the whole directory got abandoned.

A regression suite that proves both the bug and the fix

Reading and manually rerunning scenarios is how you find bugs. A test suite is how you make sure they stay fixed. This suite uses pytest’s built-in tmp_path fixture so every test gets its own throwaway directory:

import json
import os

import pytest

from usage_report import generate_usage_report


def write_log(path, lines):
    with open(path, "w", encoding="utf-8") as handle:
        handle.write("\n".join(lines) + "\n")


def test_counts_valid_levels_and_skips_malformed_lines(tmp_path):
    write_log(
        tmp_path / "app.log",
        [
            "2026-09-16T10:00:00 INFO Service started",
            "2026-09-16T10:00:01 ERROR Disk full on /var",
            "CORRUPTED LINE",
            "2026-09-16T10:00:02 WARNING High memory usage",
            "2026-09-16T10:00:03 CRITICAL Database unreachable",
        ],
    )

    result = generate_usage_report(str(tmp_path), str(tmp_path / "out.json"))

    assert result["counts"] == {
        "DEBUG": 0, "INFO": 1, "WARNING": 1, "ERROR": 1, "CRITICAL": 1,
    }
    assert result["total_lines"] == 5
    assert result["skipped_lines"] == 1
    assert result["files_processed"] == 1


def test_respects_reportignore_file(tmp_path):
    write_log(tmp_path / "app.log", ["2026-09-16T10:00:00 INFO ok"])
    write_log(tmp_path / "secrets.log", ["2026-09-16T10:01:00 ERROR should not be read"])
    (tmp_path / ".reportignore").write_text("secrets.log\n", encoding="utf-8")

    result = generate_usage_report(str(tmp_path), str(tmp_path / "out.json"))

    assert result["files_processed"] == 1
    assert result["counts"]["ERROR"] == 0
    assert result["counts"]["INFO"] == 1


def test_ignores_non_log_files(tmp_path):
    write_log(tmp_path / "app.log", ["2026-09-16T10:00:00 INFO ok"])
    (tmp_path / "notes.txt").write_text("2026-09-16T10:00:00 ERROR not a log file\n", encoding="utf-8")

    result = generate_usage_report(str(tmp_path), str(tmp_path / "out.json"))

    assert result["files_processed"] == 1
    assert result["counts"]["ERROR"] == 0


def test_output_file_matches_return_value(tmp_path):
    write_log(tmp_path / "app.log", ["2026-09-16T10:00:00 INFO ok"])
    out_path = tmp_path / "out.json"

    result = generate_usage_report(str(tmp_path), str(out_path))

    with open(out_path, "r", encoding="utf-8") as handle:
        on_disk = json.load(handle)
    assert on_disk == result
    assert os.path.getsize(out_path) > 0


def test_empty_directory_produces_zero_counts(tmp_path):
    result = generate_usage_report(str(tmp_path), str(tmp_path / "out.json"))

    assert result["files_processed"] == 0
    assert result["total_lines"] == 0
    assert all(v == 0 for v in result["counts"].values())


def test_missing_directory_raises_file_not_found(tmp_path):
    missing = tmp_path / "does-not-exist"
    with pytest.raises(FileNotFoundError):
        generate_usage_report(str(missing), str(tmp_path / "out.json"))


def test_output_file_written_inside_log_dir_is_not_read_as_input(tmp_path):
    write_log(tmp_path / "app.log", ["2026-09-16T10:00:00 INFO ok"])
    out_path = tmp_path / "out.json"  # deliberately inside log_dir

    result = generate_usage_report(str(tmp_path), str(out_path))

    assert result["files_processed"] == 1
    assert result["counts"]["INFO"] == 1

Run it against the fixed module:

$ pytest test_usage_report.py -v

test_usage_report.py::test_counts_valid_levels_and_skips_malformed_lines PASSED
test_usage_report.py::test_respects_reportignore_file PASSED
test_usage_report.py::test_ignores_non_log_files PASSED
test_usage_report.py::test_output_file_matches_return_value PASSED
test_usage_report.py::test_output_file_written_inside_log_dir_is_not_read_as_input PASSED
test_usage_report.py::test_empty_directory_produces_zero_counts PASSED
test_usage_report.py::test_missing_directory_raises_file_not_found PASSED

============================== 7 passed in 0.06s ==============================

Now point the exact same test file at the original, unedited generated code, changing nothing but the import:

$ pytest test_usage_report.py -v

test_usage_report.py::test_counts_valid_levels_and_skips_malformed_lines FAILED
test_usage_report.py::test_respects_reportignore_file FAILED
test_usage_report.py::test_ignores_non_log_files FAILED
test_usage_report.py::test_output_file_matches_return_value FAILED
test_usage_report.py::test_empty_directory_produces_zero_counts FAILED
test_usage_report.py::test_missing_directory_raises_file_not_found PASSED

ValueError: not enough values to unpack (expected 3, got 2)
assert 4 == 1
assert 3 == 1
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 65)
assert 1 == 0

========================= 5 failed, 1 passed in 0.08s ==========================

Five of six tests fail against the code exactly as it was generated. The only one that passes is the missing-directory check, which happened to already be correct. This is the concrete version of everything Step 5 found by hand: the same suite, same assertions, one true and one false, proving both the diagnosis and the cure.

Rerunning the automated checks

Finish by rerunning Step 3’s tools against the fixed file:

$ ruff check usage_report.py
All checks passed!

$ mypy usage_report.py --strict
Success: no issues found in 1 source file

$ bandit usage_report.py -q
(no output: zero issues)

Common Mistakes and Gotchas

• A clean automated-check run is not a clean bug run. Every real functional bug in this file (the ignore logic, the level-counting inversion, the crash-on-malformed-line, the missing extension filter) passed Ruff, mypy --strict, and Bandit without a single flag. Automated checks catch a different, narrower set of problems than a human reading with intent.
• A try/except block existing doesn’t mean it’s catching the right thing. Check exactly which line can raise the exception you’re guarding against, and confirm the guard actually wraps that line. This file’s ValueError handler is a textbook example of getting that placement wrong.
• No exception is not the same as correct. The scariest result in this whole exercise, Scenario 1, never raises anything. It returns a plausible-looking dict and writes an empty file. A crash gets noticed; a silent wrong answer does not.
• Check every category even when you expect it to be clean. Security and hallucinated packages came back clean here, and saying so explicitly (instead of skipping the category because nothing obvious jumped out) is what turns “I didn’t notice a problem” into “I checked, and there isn’t one.”
• Reviewer fatigue is real and it’s not about intelligence. Real Python’s own guidance puts the drop-off around four hundred lines or an hour of reading. If a diff is bigger than that, split the review into sessions instead of pushing through while your attention is already gone.

How to Confirm It All Works End to End

Rerun the full toolchain against the fixed usage_report.py in one pass and confirm every layer is green:

ruff check usage_report.py && \
mypy usage_report.py --strict && \
bandit usage_report.py -q && \
pytest test_usage_report.py -v

You should see: no Ruff errors, no mypy errors, no Bandit findings, and 7 passed from pytest. If any of those four commands produces output other than success, you have a real regression, not a false alarm, because every one of them is wired to something concrete: a style rule, a type contract, a known-insecure pattern, or a behavior the checklist from Step 2 defines.

Next Steps

This workflow catches bugs after the code exists. A few related tutorials on this site attack the same problem from other angles:

• How to Catch Vulnerable Python Code Before It’s Committed With Bandit wires exactly this kind of security scan into a git hook, so it runs automatically before a review even starts.
• How to Catch Vacuous AI-Generated Tests With Mutation Testing in Python applies the same “don’t trust it just because it runs” principle to AI-generated tests specifically, using mutation testing to check whether they’d actually catch a real bug.
• How to Write an AGENTS.md File to Guide AI Coding Agents works upstream of this tutorial: giving an agent explicit conventions and constraints before it writes anything, to reduce how many of these findings you have to catch on review.
• How to Safely Refactor Legacy Python Code Using Characterization Tests is the technique to reach for when you’re reviewing an agent’s changes to code that has no tests yet at all.