TRENDING
Vintage silver stopwatch with a chain on a black background, its dial showing seconds from 5 to 60
October 28, 2026
How to Find a Slow Query in PostgreSQL 18 With pg_stat_statements, Read Its EXPLAIN ANALYZE Plan, and Fix It With the Right Index
A wooden Tower of Hanoi puzzle on a market table, with three pegs and stacked discs of different sizes
October 28, 2026
Two Labs’ Claimed Refutations of the Exponential Time Hypothesis Turn Verification Into a Statement-Reading Problem, and Their Lean Theorems Promise Only That a Constant Exists
Two earthworms lying together on wet brown leaf litter, photographed close up
October 28, 2026
Tailscale Merged Fixes for Most of the 11 “Tailworm” Findings Before Shipping a Release, and the Forwarding Switch It Recommends Already Exists in the Shipped Client
Unpopped corn kernels mixed with popped popcorn on a white surface
October 28, 2026
How to Stop Decompression Bombs in Python With Hard Output Limits for gzip, Brotli, zstd, ZIP, and tar.gz
Ouroboros, a polished stainless steel sculpture of two interlocking loops with a pierced surface, standing beside a dark pool under trees
October 28, 2026
Anthropic’s S-1 Puts Safety Audits Inside the “Closing the Loop” Cycle It Warns About, and Says Its Evaluations Could Overstate Safety
28 Oct 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
A mountain gorilla looks at the camera from green forest undergrowth
How to Compress Time-Series Metrics in Python With Gorilla Delta-of-Delta and XOR Encoding
October 28, 2026
A bronze-framed hourglass with white sand beside a small Ferris wheel ornament on a wooden table against a teal wall
NIST’s Final Post-Quantum Transition Report Puts Every Quantum-Vulnerable Public-Key Scheme on One 2030 and 2035 Clock, and Three Federal Documents Word the Deadline Three Ways
October 28, 2026
Six weathered stone columns of the Temple of Apollo at Delphi, Greece, standing on a stone platform above a valley of olive groves
Honeypots Recorded Probes Against Oracle E-Business Suite CVE-2026-101767 Seven Days After Its Patch, and the Finder’s Write-Up Is Dated Patch Day
October 28, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 269 Posts
News 272 Posts
Learning Hub 242 Posts
Home/Learning Hub/How to Write an AGENTS.md File to Guide AI Coding Agents
Learning Hub

How to Write an AGENTS.md File to Guide AI Coding Agents

A hands-on guide to writing AGENTS.md for AI coding agents, tested end to end against a local model, including two real bugs it shipped and why a passing test suite never caught them.

August 24, 2026 20 Min Read
101

Point an AI coding agent at a real project and ask it for a small, ordinary change, and you get code back fast. Whether that code fits your project is a different question: does it use the exception type your codebase already standardizes on, does it touch a file that is supposed to be generated and never hand-edited, does it run your test suite the way your team actually runs it? A human teammate learns those rules by working with you for a few weeks. An AI agent starts every session with none of that context, unless you write it down somewhere the agent will actually read.

Table Of Content

  • What You Will Learn
  • Prerequisites
  • Step 1: Understand What AGENTS.md Actually Is
  • Step 2: Set Up a Demo Project With Real Conventions
  • Step 3: Build a Minimal Coding-Agent Harness
  • Step 4: Run the Task Without AGENTS.md
  • What Went Wrong
  • Step 5: Write Your First AGENTS.md
  • Step 6: Re-run With AGENTS.md, and See a Real Half-Fix
  • The Gap Was in the AGENTS.md, Not the Model
  • Step 7: Add an Explicit Verification Requirement
  • Fixing the Harness, Not Just the Instructions
  • Step 8: The Run That Finished, and the Two Bugs It Shipped
  • Why the Passing Tests Never Caught Either One
  • Step 9: Fix It and Verify End to End
  • Nested AGENTS.md Files for Larger Projects
  • Common Mistakes and Gotchas
  • Verify Your Own AGENTS.md Is Working
  • Next Steps

AGENTS.md is an open, plain-Markdown convention for writing that context down once, in a place dozens of different AI coding tools already know to look. This tutorial teaches you how to write one from scratch, and then proves, with real transcripts from a local model, exactly what it fixes and exactly what it does not. You will run the same ordinary task twice against a small demo project, once with no AGENTS.md and once with one, and compare the real, unedited results. Along the way the agent breaks in several distinct, genuine ways, and you will see precisely how each break happened and how to catch it. The workflow is inspired by Real Python’s How to Write an AGENTS.md File for a Python Project, which demonstrates the same idea against a small FastAPI car-catalog service, using whatever coding agent the reader already has. This version rebuilds the exercise from scratch with an original inventory-pricing project and a minimal tool-calling harness you build yourself, running entirely against a local Ollama model, so every result below is something you can reproduce and verify on your own machine.

What You Will Learn

  • What AGENTS.md actually is, who reads it, and why it is kept separate from your README
  • How to write an effective AGENTS.md, section by section, using the format the standard itself recommends
  • A real before-and-after comparison: the same task, run twice, with a genuine transcript from each run
  • Two real bugs an AI agent shipped even with a good AGENTS.md in place, and why your test suite passing did not catch either one
  • How nested AGENTS.md files work in larger projects, and how conflicting instructions get resolved

Prerequisites

  • Python 3.11 or newer, with pip available
  • Ollama installed and running locally, with a tool-calling capable model pulled. This tutorial uses qwen3.5:4b (ollama pull qwen3.5:4b), a small model chosen deliberately: it makes real mistakes, which is exactly what this tutorial needs to show you what AGENTS.md does and does not fix
  • pip install requests pytest
  • Comfortable reading Python; no prior experience with AI agent tooling required
  • Basic familiarity with git is helpful but not required

Step 1: Understand What AGENTS.md Actually Is

Straight from the standard’s own site: AGENTS.md is “a simple, open format for guiding coding agents,” used by over 60,000 open-source projects at the time of writing. The pitch is that a README is written for humans (quick starts, project descriptions, contribution guidelines), while an agent often needs different, more exhaustive information: exact build and test commands, code style rules, and gotchas that would clutter a README but that a coding agent has no way to infer on its own. Rather than every AI vendor inventing its own proprietary file, AGENTS.md is a plain Markdown file that a growing list of tools all read the same way, including OpenAI’s Codex, Google’s Jules and Gemini CLI, Cursor, Aider, Amp, Factory, JetBrains’ Junie, GitHub Copilot’s coding agent, Windsurf, and Zed. It is stewarded by the Agentic AI Foundation under the Linux Foundation.

A few specifics worth knowing before you write one, verified directly against the standard’s own FAQ:

  • There are no required fields. AGENTS.md is standard Markdown; an agent just parses whatever text you provide. The “popular choices” the standard recommends are Project overview, Build and test commands, Code style guidelines, Testing instructions, and Security considerations, and that is exactly the structure this tutorial uses below.
  • Agents will run the commands you list, if you list them. The FAQ is direct about this: “the agent will attempt to execute relevant programmatic checks and fix failures before finishing the task,” but only for commands you actually wrote down. Describing what a good test looks like is not the same instruction as telling the agent to run your tests before it finishes, a distinction that turns out to matter a great deal later in this tutorial.
  • In a large repository, the nearest AGENTS.md wins. For a monorepo, you can place a separate AGENTS.md inside each package; agents read the closest file in the directory tree to whatever they are editing, so every subproject can carry its own tailored instructions. As of this writing, OpenAI’s own main repository ships 88 separate AGENTS.md files. If instructions conflict, the closest file to the edited code takes precedence, and an explicit instruction you type in chat overrides every AGENTS.md.

The rest of this tutorial builds a small project, writes an AGENTS.md for it, and tests both the “it worked” and the “here is exactly where it did not” parts honestly.

Step 2: Set Up a Demo Project With Real Conventions

Create a project directory with a few files that establish real, non-obvious conventions, the kind an agent cannot guess just by looking at a blank folder.

mkdir -p demo_project/tests
cd demo_project

Create catalog_source.json, the human-editable source of truth for product prices:

{
  "SKU-1001": "19.99",
  "SKU-1002": "84.50",
  "SKU-1003": "5.25",
  "SKU-1004": "249.00"
}

Create generate_catalog.py, which turns that source file into a generated Python module:

"""Regenerates catalog_data.py from catalog_source.json.

Run this after editing catalog_source.json. Never hand-edit catalog_data.py;
it is overwritten every time this script runs.
"""
import json
from pathlib import Path

SOURCE = Path(__file__).parent / "catalog_source.json"
OUTPUT = Path(__file__).parent / "catalog_data.py"

HEADER = '''"""AUTO-GENERATED FILE. DO NOT EDIT BY HAND.

Regenerate with: python generate_catalog.py
Source of truth: catalog_source.json
"""

CATALOG = {
'''


def main() -> None:
    source = json.loads(SOURCE.read_text(encoding="utf-8"))
    lines = [HEADER]
    for sku, price in sorted(source.items()):
        lines.append(f'    "{sku}": "{price}",\n')
    lines.append("}\n")
    OUTPUT.write_text("".join(lines), encoding="utf-8")
    print(f"Wrote {OUTPUT} with {len(source)} SKUs.")


if __name__ == "__main__":
    main()

Run it once to generate catalog_data.py:

python generate_catalog.py

Expected output:

Wrote /path/to/demo_project/catalog_data.py with 4 SKUs.

Now the actual business logic, inventory.py, which establishes two conventions on its own: money is always Decimal, never float, and every domain-level error raises a custom InventoryError instead of a built-in exception:

"""Core pricing logic for the demo inventory service.

All money values are Decimal, never float, so totals never pick up
floating-point rounding error. Domain errors raise InventoryError so
callers can catch pricing problems separately from programming bugs.
"""
from decimal import Decimal, ROUND_HALF_UP

from catalog_data import CATALOG


class InventoryError(Exception):
    """Raised for domain-level pricing problems (bad SKU, bad quantity)."""


def get_item_price(sku: str) -> Decimal:
    """Look up the unit price for a SKU.

    Raises:
        InventoryError: If the SKU is not in the catalog.
    """
    if sku not in CATALOG:
        raise InventoryError(f"Unknown SKU: {sku}")
    return Decimal(CATALOG[sku])


def order_total(sku: str, quantity: int) -> Decimal:
    """Compute the total price for a quantity of one SKU.

    Raises:
        InventoryError: If the SKU is unknown or quantity is not positive.
    """
    if quantity <= 0:
        raise InventoryError(f"Quantity must be positive, got {quantity}")
    unit_price = get_item_price(sku)
    total = unit_price * quantity
    return total.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

And cli.py, a tiny command-line front end:

"""Command-line entry point for the demo inventory service."""
import argparse
import sys

from inventory import InventoryError, order_total


def main() -> None:
    parser = argparse.ArgumentParser(prog="cli.py")
    sub = parser.add_subparsers(dest="command", required=True)

    total_cmd = sub.add_parser("total", help="Price an order for one SKU")
    total_cmd.add_argument("sku")
    total_cmd.add_argument("quantity", type=int)

    args = parser.parse_args()

    if args.command == "total":
        try:
            total = order_total(args.sku, args.quantity)
        except InventoryError as exc:
            print(f"Error: {exc}", file=sys.stderr)
            sys.exit(1)
        print(f"Total for {args.quantity}x {args.sku}: ${total}")


if __name__ == "__main__":
    main()

Finally, register a custom pytest marker in pytest.ini:

[pytest]
markers =
    slow: marks tests that hit the simulated remote pricing feed (deselect with '-m "not slow"')

And tests/test_inventory.py, with one test deliberately marked slow to simulate a real integration test that hits a remote system:

"""Tests for inventory.py.

Run the fast suite during normal development: pytest -m "not slow"
The slow tests simulate hitting the remote pricing feed and are only
meant for the pre-release check, not the inner dev loop.
"""
import time
from decimal import Decimal

import pytest

from inventory import InventoryError, get_item_price, order_total


def test_get_item_price_known_sku():
    assert get_item_price("SKU-1001") == Decimal("19.99")


def test_get_item_price_unknown_sku_raises():
    with pytest.raises(InventoryError):
        get_item_price("SKU-9999")


def test_order_total_multiplies_and_rounds():
    assert order_total("SKU-1003", 3) == Decimal("15.75")


def test_order_total_rejects_zero_quantity():
    with pytest.raises(InventoryError):
        order_total("SKU-1001", 0)


@pytest.mark.slow
def test_order_total_matches_remote_pricing_feed():
    """Simulates a round trip to a remote pricing feed for reconciliation."""
    time.sleep(3)
    assert order_total("SKU-1002", 2) == Decimal("169.00")

Confirm the project actually works before you hand it to an agent. This project has a real, timeable convention: the full suite takes seconds longer than the filtered one, because of that one slow test.

python -m pytest -v
collected 5 items
tests/test_inventory.py::test_get_item_price_known_sku PASSED
tests/test_inventory.py::test_get_item_price_unknown_sku_raises PASSED
tests/test_inventory.py::test_order_total_multiplies_and_rounds PASSED
tests/test_inventory.py::test_order_total_rejects_zero_quantity PASSED
tests/test_inventory.py::test_order_total_matches_remote_pricing_feed PASSED
5 passed in 3.06s
python -m pytest -m "not slow" -v
collected 5 items / 1 deselected / 4 selected
tests/test_inventory.py::test_get_item_price_known_sku PASSED
tests/test_inventory.py::test_get_item_price_unknown_sku_raises PASSED
tests/test_inventory.py::test_order_total_multiplies_and_rounds PASSED
tests/test_inventory.py::test_order_total_rejects_zero_quantity PASSED
4 passed, 1 deselected in 0.02s

Both real, timed runs: 3.06 seconds with the slow test included, 0.02 seconds (0.24 seconds wall clock, including Python startup) filtered out. That gap is the entire reason the project has a documented “fast suite for daily work, full suite before a pull request” convention, exactly the kind of unwritten rule an agent has no way to know unless you tell it.

This tutorial runs the same task against this project four times below, so save a clean baseline to reset from before you hand it to an agent. The simplest way, if you have git installed:

git init
git add -A
git commit -m "baseline"

Reset back to this exact state before each run in the rest of this tutorial with:

git checkout -- .
git clean -fd

Without git, copying the whole demo_project folder to a backup location and restoring it before each run works just as well.

Step 3: Build a Minimal Coding-Agent Harness

To test AGENTS.md honestly, you need something that behaves mechanically like a real coding agent: it reads files, writes files, runs a command, and decides for itself when a task is done. Commercial tools hide this loop behind a UI, so this tutorial builds a minimal, transparent version of the same loop against Ollama’s chat API with tool calling. Like a real coding agent, it looks for an AGENTS.md file in the project root before starting and, if one exists, feeds its contents into the system prompt.

"""Minimal local tool-calling coding-agent harness for the AGENTS.md demo."""
import json
import subprocess
import sys
from pathlib import Path

import requests

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "qwen3.5:4b"
MAX_STEPS = 14

TOOLS = [
    {"type": "function", "function": {
        "name": "list_files",
        "description": "List all files in the project (relative paths).",
        "parameters": {"type": "object", "properties": {}, "required": []},
    }},
    {"type": "function", "function": {
        "name": "read_file",
        "description": "Read the full contents of a file in the project.",
        "parameters": {"type": "object", "properties": {
            "path": {"type": "string", "description": "Relative path, e.g. inventory.py"}
        }, "required": ["path"]},
    }},
    {"type": "function", "function": {
        "name": "write_file",
        "description": "Overwrite a file in the project with new full contents.",
        "parameters": {"type": "object", "properties": {
            "path": {"type": "string"},
            "content": {"type": "string"},
        }, "required": ["path", "content"]},
    }},
    {"type": "function", "function": {
        "name": "run_tests",
        "description": "Run the project's test suite with pytest. args is appended, e.g. '-m \"not slow\"'.",
        "parameters": {"type": "object", "properties": {
            "args": {"type": "string"}
        }, "required": ["args"]},
    }},
    {"type": "function", "function": {
        "name": "task_complete",
        "description": "Call this when the task is fully done, with a short summary of what you changed.",
        "parameters": {"type": "object", "properties": {
            "summary": {"type": "string"}
        }, "required": ["summary"]},
    }},
]


def list_files(project_dir: Path) -> str:
    files = sorted(
        str(p.relative_to(project_dir)).replace("\\", "/")
        for p in project_dir.rglob("*")
        if p.is_file() and ".git" not in p.parts
        and "__pycache__" not in p.parts and ".pytest_cache" not in p.parts
    )
    return "\n".join(files)


def read_file(project_dir: Path, path: str) -> str:
    target = (project_dir / path).resolve()
    if not target.exists():
        return f"ERROR: file not found: {path}"
    return target.read_text(encoding="utf-8")


def write_file(project_dir: Path, path: str, content: str) -> str:
    target = (project_dir / path).resolve()
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(content, encoding="utf-8")
    return f"OK: wrote {len(content)} bytes to {path}"


def run_tests(project_dir: Path, args: str) -> str:
    cmd = [sys.executable, "-m", "pytest"]
    if args.strip():
        cmd.extend(args.split())
    proc = subprocess.run(cmd, cwd=project_dir, capture_output=True, text=True, timeout=60)
    return f"exit_code={proc.returncode}\n{(proc.stdout + proc.stderr)[-4000:]}"


def dispatch(project_dir: Path, name: str, args: dict) -> str:
    if name == "list_files":
        return list_files(project_dir)
    if name == "read_file":
        return read_file(project_dir, args.get("path", ""))
    if name == "write_file":
        return write_file(project_dir, args.get("path", ""), args.get("content", ""))
    if name == "run_tests":
        return run_tests(project_dir, args.get("args", ""))
    return f"ERROR: unknown tool {name}"


def run_agent(project_dir: Path, task_prompt: str) -> None:
    agents_md = project_dir / "AGENTS.md"
    system_prompt = (
        "You are a coding agent working inside the project directory. "
        "Use the provided tools to inspect and modify files and to run tests. "
        "Call task_complete with a short summary once the task is fully done. "
        "Do not call task_complete until you have actually written the code."
    )
    if agents_md.exists():
        system_prompt += "\n\nAGENTS.md found in the project root:\n\n" + agents_md.read_text(encoding="utf-8")

    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": task_prompt},
    ]

    for step in range(MAX_STEPS):
        resp = requests.post(OLLAMA_URL, json={
            "model": MODEL, "messages": messages, "tools": TOOLS,
            "think": False, "stream": False,
        }, timeout=180)
        resp.raise_for_status()
        message = resp.json()["message"]
        messages.append(message)

        tool_calls = message.get("tool_calls") or []
        if not tool_calls:
            break

        done = False
        for call in tool_calls:
            fn = call["function"]["name"]
            raw_args = call["function"]["arguments"]
            args = raw_args if isinstance(raw_args, dict) else json.loads(raw_args)
            if fn == "task_complete":
                done = True
                continue
            result = dispatch(project_dir, fn, args)
            messages.append({"role": "tool", "content": result})
        if done:
            break


if __name__ == "__main__":
    run_agent(Path(sys.argv[1]).resolve(), Path(sys.argv[2]).read_text(encoding="utf-8"))

This is a genuine tool-calling loop, not a simulation: every read_file, write_file, and run_tests call below executes for real against the files on disk. The one honest shortcut is think: False, which turns off this particular model’s chain-of-thought mode. Without it, qwen3.5:4b spends most of its response budget on hidden reasoning tokens before ever reaching a tool call, which is a real, worth-knowing quirk of reasoning models in tool-calling loops, not specific to this tutorial.

Write the task you will hand the agent, twice, to task_prompt.txt. It is deliberately the kind of ordinary request a real project gets constantly:

Add a function to inventory.py that applies a percentage discount to an
order total, plus a new "discount" CLI command in cli.py that takes a SKU,
a quantity, and a discount percent, and prints the discounted total. Add a
test for the new function too.

Step 4: Run the Task Without AGENTS.md

With no AGENTS.md in the project directory, run the harness:

python agent_harness.py demo_project task_prompt.txt

Here is the real, captured sequence of tool calls from that run: the agent read inventory.py and cli.py first, wrote a new apply_discount function and a new cli.py, discovered pytest.ini and the test file on its own, added five tests, and ran the suite before declaring the task complete.

read_file(inventory.py)
read_file(cli.py)
write_file(inventory.py)        # adds apply_discount
write_file(cli.py)              # adds the discount subcommand
read_file(pytest.ini)
list_files()
read_file(tests/test_inventory.py)
write_file(tests/test_inventory.py)  # adds 5 new tests
run_tests(args="")
"All tests pass. Here's a summary of the changes..."

What Went Wrong

The test run really did pass, 9 of 9. But two real problems are hiding in a clean test run:

First, even though the agent had just read inventory.py and seen every existing function raise InventoryError, its own new validation code raised a plain ValueError instead:

# existing code the agent had just read
raise InventoryError(f"Unknown SKU: {sku}")
raise InventoryError(f"Quantity must be positive, got {quantity}")

# the agent's own new code, one function below
raise ValueError(f"Discount must be between 0 and 100, got {discount_percent}")

That is a real, working inconsistency: any caller written to catch InventoryError and handle every pricing problem in one place will let this particular one through uncaught. Reading code is not the same as internalizing an unstated convention, and a model will happily do both in the same file.

Second, the agent ran the full test suite with a bare pytest, the slow, 3-second version, not the pytest -m "not slow" the project’s own docstrings describe as the normal development command. Nothing failed, so nothing looked wrong, but the agent never learned the project has a fast path at all.

Neither of these is a dramatic failure. That is the point: AGENTS.md is not about stopping catastrophes, it is about the much more common case of a technically-working change that quietly does not match how your project actually works.

Step 5: Write Your First AGENTS.md

Reset the project to its starting state, then add an AGENTS.md in the project root using the standard’s own recommended sections:

# AGENTS.md

## Project overview
A small demo inventory/pricing service. `inventory.py` holds the pricing
logic, `cli.py` is the command-line entry point, `catalog_data.py` is
generated product data.

## Build and test commands
- Run the fast test suite during normal development: `pytest -m "not slow"`
- Run the full suite (including the slow remote-pricing-feed check) before
  opening a pull request: `pytest`
- Regenerate the product catalog after editing `catalog_source.json`:
  `python generate_catalog.py`

## Code style guidelines
- All money values are `Decimal`, never `float`.
- All domain-level errors (bad SKU, bad quantity, bad discount, any input
  validation failure) must raise `InventoryError` from `inventory.py`, not a
  built-in exception like `ValueError`. Callers rely on catching
  `InventoryError` alone to handle every pricing problem.
- Every public function needs a full type-hinted signature and a
  Google-style docstring (Args/Returns/Raises).

## Testing instructions
- New pricing logic needs at least one passing-case test and one test for
  each `InventoryError` condition it can raise.
- Mark any test that simulates a remote/network round trip with
  `@pytest.mark.slow` so `pytest -m "not slow"` stays fast.

## Security considerations
- `catalog_data.py` is auto-generated. Never edit it by hand; edit
  `catalog_source.json` and run `python generate_catalog.py` instead.

Each section maps directly to something the previous run got wrong or never learned: Code style guidelines names the exact exception rule that slipped through, Build and test commands states the fast-suite convention explicitly instead of leaving it to be discovered, and Security considerations calls out the one file that must never be hand-edited, which this run never actually touched but a less careful model easily could.

Step 6: Re-run With AGENTS.md, and See a Real Half-Fix

Run the identical task prompt again, from a freshly reset project, with AGENTS.md now in place:

python agent_harness.py demo_project task_prompt.txt

The improvement is real and immediate. Compare the agent’s new code directly against the Step 4 run: this time inventory.py raised InventoryError consistently, including for the new discount check, never once reaching for a plain ValueError. That is direct, working evidence the Code style guidelines section did exactly what it was written to do. (Neither run touched catalog_data.py, so this particular task never actually put the Security considerations section to the test; a task that tempted the agent to edit generated data would be a better check of that rule specifically.)

But the run did not finish cleanly. On its last write to tests/test_inventory.py, the model produced genuinely broken Python: a stray parametrized test containing literal backslash-n text instead of a real line break, and a call to an undefined gettype() function. Running pytest against it failed at collection, not at a test:

SyntaxError: unexpected character after line continuation character

The model’s own next message even said, “The file got corrupted. Let me fix it properly,” but it ended the turn without calling any tool to actually fix it, so the harness’s loop ended with the file still broken. The task was never verified, because it was never actually finished, and nothing in the transcript announced that clearly.

The Gap Was in the AGENTS.md, Not the Model

Look again at the Testing instructions section from Step 5. It describes what tests should exist. It never tells the agent to run them before declaring the task done. That distinction matters: the standard’s own FAQ says an agent will run the commands you list and try to fix failures “before finishing the task,” but only for commands you actually wrote down as an instruction, not test-writing advice you happened to mention nearby. A first-draft AGENTS.md that only describes good tests, without requiring a passing run of them, leaves exactly this gap.

Step 7: Add an Explicit Verification Requirement

Fix the Testing instructions section with an explicit, unambiguous requirement:

## Testing instructions
- New pricing logic needs at least one passing-case test and one test for
  each `InventoryError` condition it can raise.
- Mark any test that simulates a remote/network round trip with
  `@pytest.mark.slow` so `pytest -m "not slow"` stays fast.
- Before finishing, run `pytest -m "not slow"` and make sure it exits 0. If
  it fails, fix the code or the test and run it again. Do not report the
  task as complete on a failing or unverified test run.

Reset the project again and re-run the identical prompt. This time a different, equally real problem shows up: the model’s final message says “Now let me write tests for the new function,” then the turn ends with no tool call at all, exactly like the previous run’s “let me fix it” moment. The task stops again, tests unwritten, tests unrun.

That is not a fluke of one bad response. It is a real, known limitation of a minimal single-shot tool-calling loop: a small model can narrate an intention in plain text without emitting the tool call in the same turn, and a harness that treats “no tool call” as “finished” will end the session right when the model was about to keep working.

Fixing the Harness, Not Just the Instructions

The fix is a standard technique: if the model responds with no tool call and has not called task_complete, nudge it to continue instead of ending the loop, up to a small budget of attempts so a genuinely stuck model does not loop forever.

nudges_left = 3

for step in range(MAX_STEPS):
    # ... same request as before ...
    tool_calls = message.get("tool_calls") or []
    if not tool_calls:
        if nudges_left > 0:
            nudges_left -= 1
            messages.append({"role": "user", "content":
                "Continue. If the task is not finished yet, keep using "
                "the tools until it is, then call task_complete."})
            continue
        break  # out of nudges, stop for real

Step 8: The Run That Finished, and the Two Bugs It Shipped

Re-run once more, same reset project, same AGENTS.md, now with the nudge-capable harness. This run used all three nudges: the agent rewrote cli.py three times in a row, went quiet for three straight turns despite the nudges, made one more file read, and then stopped for good, still without writing a test or calling run_tests or task_complete even once.

This is the most important result in this tutorial, so read the actual final code carefully rather than trusting the transcript. The exception-handling convention held up again: every new validation raised InventoryError, never ValueError. But two real, silent bugs shipped anyway.

Bug one, in the discount math itself:

# the agent's actual output
discounted = order_total * Decimal("0.01") - order_total * (Decimal(str(discount_percent)) / Decimal("100"))

Calling this directly exposes it immediately:

>>> apply_discount(Decimal("100"), 10)
Decimal('-9.00')     # a 10% discount on $100 should be $90.00
>>> apply_discount(Decimal("157.50"), 20)
Decimal('-29.93')    # a 20% discount on $157.50 should be $126.00

Both real results are negative. The correct formula is total * (1 - percent / 100); the model’s version instead computed total * 0.01 - total * (percent / 100), which only produces a small positive number when the percentage is under 1.

Bug two is even quieter. The agent’s rewritten cli.py never restored the script’s own entry point:

if __name__ == "__main__":
    main()

Across its three rewrites of the file, that block simply never made it back in. main() was still defined correctly, argument parsing and all, but nothing in the file ever called it when run directly:

$ python cli.py discount SKU-1001 3 10
$ echo $?
0

No output. No error. Exit code 0. A command that silently does nothing at all is arguably worse than one that crashes, because nothing about running it tells you anything is wrong.

Why the Passing Tests Never Caught Either One

This is the sharpest lesson in the whole exercise. Run the project’s own test suite against this exact broken state:

python -m pytest -m "not slow" -v
4 passed, 1 deselected in 0.03s

Green. Every test passes, because every test that exists is one of the four original tests from Step 2; no test for apply_discount was ever written, so nothing exercises the broken formula. A green test suite only tells you that the tests you have still pass, not that the feature you asked for actually works. And unit tests that import apply_discount and call it directly, which is exactly how you would normally test it, never touch the if __name__ == "__main__": guard at all, so that class of bug is invisible to unit tests by construction; you have to actually run the command-line entry point to catch it.

Step 9: Fix It and Verify End to End

This is the step a developer reviewing the agent’s change would do next regardless of what any AGENTS.md says: read the diff, run the thing, and fix what is wrong. Correct the formula:

discounted = order_total * (Decimal("1") - Decimal(str(discount_percent)) / Decimal("100"))

Restore the missing entry point and wire the discount function into the CLI branch, which had been left computing a total and then doing nothing with it:

    elif args.command == "discount":
        try:
            total = order_total(args.sku, args.quantity)
            discounted = apply_discount(total, args.percent)
        except InventoryError as exc:
            print(f"Error: {exc}", file=sys.stderr)
            sys.exit(1)
        print(f"Total for {args.quantity}x {args.sku}: ${total}")
        print(f"With {args.percent}% discount: ${discounted}")


if __name__ == "__main__":
    main()

Add the test that should have existed from the start:

def test_apply_discount_applies_correct_rate():
    assert apply_discount(Decimal("157.50"), 20) == Decimal("126.00")


def test_apply_discount_zero_percent_returns_original():
    assert apply_discount(Decimal("50.00"), 0) == Decimal("50.00")


def test_apply_discount_rejects_over_100_percent():
    with pytest.raises(InventoryError):
        apply_discount(Decimal("50.00"), 101)

Now verify for real, both the automated suite and the actual command a user would type:

python -m pytest -v
tests/test_inventory.py::test_get_item_price_known_sku PASSED
tests/test_inventory.py::test_get_item_price_unknown_sku_raises PASSED
tests/test_inventory.py::test_order_total_multiplies_and_rounds PASSED
tests/test_inventory.py::test_order_total_rejects_zero_quantity PASSED
tests/test_inventory.py::test_order_total_matches_remote_pricing_feed PASSED
tests/test_inventory.py::test_apply_discount_applies_correct_rate PASSED
tests/test_inventory.py::test_apply_discount_zero_percent_returns_original PASSED
tests/test_inventory.py::test_apply_discount_rejects_over_100_percent PASSED
8 passed in 3.04s
$ python cli.py discount SKU-1001 3 10
Total for 3x SKU-1001: $59.97
With 10% discount: $53.97

3 units of a $19.99 item is $59.97, and 10% off that is $53.97: real, correct output from the actual command line, not just from a unit test.

Nested AGENTS.md Files for Larger Projects

The single-file setup above is enough for a small project, but the same convention scales to a monorepo without inventing anything new. Place another AGENTS.md inside any subproject that needs its own rules, a different language, a different test runner, different style guidelines, and coding agents automatically read the nearest AGENTS.md file to whatever they are editing, walking up the directory tree from there. The closest file always takes precedence over a more general one further up, and an explicit instruction you type directly in a chat prompt overrides every AGENTS.md, close or far. This is not a hypothetical scale: OpenAI’s own main repository currently ships 88 separate AGENTS.md files across its subprojects, each scoped to the code around it. If you are migrating an existing convention file, the standard recommends renaming it to AGENTS.md and adding a symbolic link back to the old filename, so any tooling still looking for the original name keeps working.

If you specifically use Claude Code, its own memory file works a little differently. See How to Write and Organize CLAUDE.md Files for Claude Code for that tool’s own conventions; a one-line @AGENTS.md import at the top of a CLAUDE.md file lets both formats share the same source of truth instead of maintaining two documents by hand.

Common Mistakes and Gotchas

  • Describing tests without requiring them to run. A Testing instructions section that only says what a good test looks like will not make an agent run your suite before finishing. Say so explicitly, and say what a failing run should not be allowed to end in.
  • Trusting silence as success. A model that stops calling tools has not necessarily finished. Watch for a real task_complete call (or your tool’s equivalent signal), not just the absence of further activity.
  • Treating a green test suite as proof. Tests only check what they were written to check. New behavior needs a new test, or a passing run proves nothing about it.
  • Forgetting unit tests cannot see your entry point. A function-level test suite will never catch a broken or missing if __name__ == "__main__": block, a broken CLI wiring, or anything else that only exists at the executable-script level. Run the actual command periodically, not just the test suite.
  • Writing AGENTS.md once and never touching it again. Treat it as living documentation. When you catch an agent making the same mistake twice, that is a sign the instructions need to be more specific, not that the agent needs to be smarter.
  • Vague instructions in a monorepo. “Use good style” tells an agent nothing actionable. Name the exact exception class, the exact command, the exact file it must never touch, the same way you would brief a new hire on their first day rather than handing them a slogan.

Verify Your Own AGENTS.md Is Working

The same technique this tutorial used to test its own AGENTS.md works on any real project: pick one ordinary task, run it with an agent before you have an AGENTS.md, and read the resulting diff for anything that violates a convention you care about. Write those violations directly into AGENTS.md as specific rules. Reset and run the identical task again. Confirm two things separately: that the new run’s diff actually follows the rule you just wrote, and that your own test suite (plus, if relevant, the command actually running) still genuinely passes rather than assuming a quiet agent means a finished one.

Next Steps

  • Add an AGENTS.md to a real project you maintain, starting with the two or three conventions an agent is most likely to violate: your test command, your money or ID types, and any file that must never be hand-edited.
  • If your project is a monorepo, add a nested AGENTS.md to your most unusual subproject first, the one whose rules differ most from everything else in the repository.
  • Whatever coding agent you actually use day to day, check whether it is on the standard’s supported tools list, then repeat the before-and-after test in this tutorial against your own project and your own real task.

Tags:

AGENTS.mdAI Coding AgentsOllamaPython

Share

A row of network and server racks with cable management running down a raised-floor data center aisle
Previous Post

Nvidia’s First Groq 3 LPX Benchmark Turns a Best-Case Demo Into a Scaling Question

The Alabama State Capitol building in Montgomery, Alabama
Next Post

Alabama’s Attorney General Subpoenas OpenAI Over the Hugging Face Breach

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
28 Oct
How to Compress Time-Series Metrics in Python With Gorilla Delta-of-Delta and XOR Encoding
28 Oct
NIST’s Final Post-Quantum Transition Report Puts Every Quantum-Vulnerable Public-Key Scheme on One 2030 and 2035 Clock, and Three Federal Documents Word the Deadline Three Ways
Trending
October 28, 2026
How to Compress Time-Series Metrics in Python With Gorilla Delta-of-Delta and XOR Encoding
October 28, 2026
NIST’s Final Post-Quantum Transition Report Puts Every Quantum-Vulnerable Public-Key Scheme on One 2030 and 2035 Clock, and Three Federal Documents Word the Deadline Three Ways
October 28, 2026
Honeypots Recorded Probes Against Oracle E-Business Suite CVE-2026-101767 Seven Days After Its Patch, and the Finder’s Write-Up Is Dated Patch Day
October 27, 2026
How to Find a Slow Query in PostgreSQL 18 With pg_stat_statements, Read Its EXPLAIN ANALYZE Plan, and Fix It With the Right Index
October 27, 2026
Two Labs’ Claimed Refutations of the Exponential Time Hypothesis Turn Verification Into a Statement-Reading Problem, and Their Lean Theorems Promise Only That a Constant Exists
October 27, 2026
Tailscale Merged Fixes for Most of the 11 “Tailworm” Findings Before Shipping a Release, and the Forwarding Switch It Recommends Already Exists in the Shipped Client

Related Posts

A laptop wrapped in a chain and padlock, illustrating least-privilege controls for AI agents.
Learning Hub

How to Secure Tool-Using AI Agents Before They Touch Production

June 8, 2026
Colorful sticky notes arranged on an office wall, symbolizing governance checklists and planning.
Learning Hub

AI Governance for Agentic Apps: A Practical Checklist for Builders

June 8, 2026
A technician connects green fiber optic cables at a data center, representing a private production inference endpoint.
Learning Hub

How to Deploy a Fine-Tuned LLM Behind a Private Production Inference Endpoint

June 8, 2026
Narrow aisle behind black supercomputer racks in a data center
Learning Hub

Kubernetes SELinux Volume Labeling: What Cluster Operators Should Audit Before v1.37

June 8, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026