How to Build a Local LLM-as-a-Judge Evaluation Harness for AI Agents

Building an AI agent is the easy part now. Most tool-calling agents work fine on the two or three questions you tried while building them. The real risk shows up later: you tweak a system prompt, swap in a cheaper model, or add a new tool, and something that used to work quietly breaks. Nobody notices until a user does.

Regular software has unit tests for exactly this problem. AI agents rarely get the same treatment, because their output is prose rather than a fixed return value, so a plain equality check does not work. A recent freeCodeCamp tutorial by Darsh Shah lays out a practical pattern for closing that gap: a small, repeatable evaluation harness that runs an agent against a fixed set of test cases and grades each answer with a mix of deterministic rule checks and a second model acting as a judge. This tutorial builds that pattern yourself, end to end, in Python, with your own tools, your own test cases, and a fully local model stack through Ollama, so there is no API key to buy and no per-call bill. Every command, package version, and piece of output below was actually run while writing this tutorial; see the “Verifying everything works end to end” section for exactly what that means.

Three terms are worth pinning down before the first command:

Agent evaluation is the practice of running an agent against a fixed list of inputs and checking whether the outputs still meet expectations: the AI equivalent of a regression test suite. The goal is not to prove the agent is perfect. The goal is to catch the moment a change makes it worse.

Rule-based check is an assertion you write in plain code: does the answer contain a certain word, did the agent call a specific tool, is a number in range. Rule-based checks are fast, free, and deterministic, but they only work for things you can phrase as a fixed rule.

LLM-as-a-judge is a second model call that reads the question, the agent’s answer, and a short grading rubric, then replies with a pass or fail verdict. It is useful for fuzzy judgments a plain rule cannot express, such as “did this answer actually address what was asked,” at the cost of being an imperfect, occasionally wrong grader itself.

What you need before you start

• A Linux machine with a terminal. This tutorial was written and tested on Ubuntu 26.04 LTS; other recent Ubuntu or Debian releases behave the same way, and macOS and Windows work too, since Ollama and Python both ship native installers for those platforms (use the matching installer from ollama.com/download and adjust paths accordingly).
• Python 3.10 or newer. Check yours with python3 --version.
• About 4 GB of free disk space for Ollama and the model this tutorial uses, and ideally 8 GB of RAM. Inference runs fine on CPU only, just slower per answer than with a GPU; the real timings later in this tutorial were measured on a CPU-only host.
• A regular user account with sudo access, needed once to install Ollama. The rest of this tutorial never needs root.
• Comfort with basic command-line use and reading simple Python. No prior experience with LangChain or agent frameworks is assumed; both are introduced from scratch below.

Step 1: Install Ollama and pull a model

Ollama is an open source runtime that downloads, serves, and runs large language models entirely on your own machine, exposing them to your code over a local API. Because both the agent and the judge in this tutorial call a model that runs on your own hardware, there is no external API key to manage and no per-request bill, only your own CPU or GPU time.

Install it with the official install script, verified against Ollama’s own Linux download documentation:

# Context: any Linux machine (Ubuntu 26.04 LTS used in this tutorial) with sudo available.
# Purpose: install the Ollama runtime and register it as a background service.
curl -fsSL https://ollama.com/install.sh | sh

Expected output: a short download progress display followed by lines confirming the Ollama service was installed and started, ending with a note telling you to run ollama from the command line. On a machine with no NVIDIA or AMD GPU you will also see a message that Ollama will run in CPU-only mode; that is expected on most laptops, desktops, and virtual machines, and is exactly the setup this tutorial was tested on.

Next, pull the model this tutorial uses for both the agent under test and the judge:

# Context: Ollama installed and running from the previous command.
# Purpose: download the 4-billion-parameter tag of Qwen 3.5 for local inference.
ollama pull qwen3.5:4b

Expected output: a “pulling manifest” line, then one progress bar per model layer counting up to 100% (the largest layer on this tag is about 3.4 GB), followed by verifying sha256 digest, writing manifest, and finally success. This is the exact sequence this tutorial produced when re-running the pull to confirm it. If your machine has less RAM, you can substitute a smaller tag such as qwen3.5:0.8b everywhere below; expect noisier, less consistent judge verdicts at that size, since a smaller model is also a weaker grader.

Step 2: Set up your Python project

Create a project folder, an isolated virtual environment, and install the three packages this tutorial needs:

# Context: a fresh project directory, Python 3.10+ already installed.
# Purpose: isolate this project's dependencies from anything else on your machine.
mkdir judge-harness && cd judge-harness
python3 -m venv venv
source venv/bin/activate
pip install langchain langchain-core langchain-ollama

Expected output: pip resolving and installing LangChain and its dependencies, ending in a Successfully installed ... line. This tutorial was tested against langchain 1.3.14, langchain-core 1.4.9, and langchain-ollama 1.1.0. The exact patch versions you get may be newer, but you need langchain>=1.0.0 at minimum: create_agent, used in the next step, is a LangChain v1 API and does not exist in older 0.x releases.

Confirm the install worked before moving on:

python3 -c "from langchain.agents import create_agent; print('ok')"

Expected output: a single line reading ok. If you instead see ModuleNotFoundError, the most common cause is forgetting to activate the virtual environment in the current shell; run source venv/bin/activate again and retry.

Step 3: Build the agent under test

The eval harness you are about to build treats the agent as an opaque system: it sends in a question and looks at what comes back, without caring how the agent produces its answer. That means you can point this same harness at any LangChain agent later. For this tutorial, build a small one with two tools, so you have something concrete to evaluate.

Save the following as agent.py:

from datetime import datetime
from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_ollama import ChatOllama


@tool
def current_time() -> str:
    """Return the current local date and time."""
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")


@tool
def count_vowels(text: str) -> int:
    """Count the number of vowels (a, e, i, o, u) in a piece of text."""
    return sum(1 for ch in text.lower() if ch in "aeiou")


def build_agent():
    model = ChatOllama(model="qwen3.5:4b", temperature=0)
    return create_agent(
        model=model,
        tools=[current_time, count_vowels],
        system_prompt="You are a helpful assistant with access to tools. Use a tool whenever the question calls for it.",
    )

A few things worth understanding line by line, since each piece reappears in every LangChain agent you will ever build:

The @tool decorator, imported from langchain_core.tools, turns an ordinary Python function into something the model is allowed to call. The function’s docstring is not a comment for humans; LangChain sends it to the model as the tool’s description, so the model can decide when the tool is relevant. Keep it short and accurate, since a vague or misleading docstring leads the model to call the wrong tool, or the right tool at the wrong time.

ChatOllama, imported from langchain_ollama, is the LangChain wrapper around a locally running Ollama model. Setting temperature=0 tells the model to always pick its highest-probability next token instead of sampling randomly. This will not make output byte-for-byte identical across runs (tool results like current_time() legitimately change every second, and model inference has some irreducible nondeterminism even at temperature 0), but it removes one large source of run-to-run variance, which matters a lot once you are relying on this same model to grade pass or fail later.

create_agent, imported from langchain.agents, wires a model and a list of tools into a working tool-calling agent in one call. The system_prompt argument is the instruction the agent sees before any user message; it is your main lever for steering behavior, such as telling the agent to actually use its tools instead of guessing.

Try it once by hand before building anything else on top of it:

python3 -c "
from agent import build_agent
agent = build_agent()
result = agent.invoke({'messages': [{'role': 'user', 'content': 'How many vowels are in the word Kubernetes?'}]})
print(result['messages'][-1].content)
"

Expected output: here is the real, unedited answer this exact command produced while writing this tutorial:

There are **4** vowels in the word "Kubernetes".

The vowels are: u, e, e, and e.

That is correct: K-u-b-e-r-n-e-t-e-s contains four vowels. If your answer differs, do not worry yet, small models occasionally get arithmetic-flavored questions wrong even with a tool available; that exact failure mode is the whole reason the next two steps exist.

Step 4: Write the eval harness

With a working agent in hand, the next job is a script that asks it a fixed list of questions and grades every answer the same way, every time.

Define the test cases

Each test case pairs one input with three independent expectations: a substring that must appear somewhere in the answer, the name of the tool the agent should have called (or None if no tool call is expected at all), and a plain-language rubric for the judge model to grade against. Start a new file, eval_agent.py, with this list:

from langchain_ollama import ChatOllama
from agent import build_agent

TEST_CASES = [
    {
        "input": "What time is it right now?",
        "expected_substring": ":",
        "expected_tool": "current_time",
        "judge_rubric": "The answer should state a specific clock time.",
    },
    {
        "input": "How many vowels are in the word Kubernetes?",
        "expected_substring": "4",
        "expected_tool": "count_vowels",
        "judge_rubric": "The answer should say the vowel count is 4.",
    },
    {
        "input": "What is the capital of Japan?",
        "expected_substring": "Tokyo",
        "expected_tool": None,
        "judge_rubric": "The answer should say Tokyo.",
    },
]

judge = ChatOllama(model="qwen3.5:4b", temperature=0)

Notice the third case expects no tool call at all: “capital of Japan” is general knowledge the model already has, and a well-behaved agent should answer it directly instead of reaching for a tool it does not need. Testing for the absence of a tool call is just as useful as testing for its presence; an agent that calls tools when it should not is burning latency and, in a paid-API setup, money, on every unnecessary call.

The two rule-based checks

Add these two functions below the test cases. Both are plain Python, no model involved, which is exactly why they are fast and perfectly repeatable:

def check_substring(answer, expected):
    return expected.lower() in answer.lower()


def check_tool(tool_calls, expected_tool):
    if expected_tool is None:
        return len(tool_calls) == 0
    return expected_tool in tool_calls

check_substring does a case-insensitive containment check, deliberately loose rather than an exact match, since a model can phrase “Tokyo” inside many different correct sentences. check_tool compares against the list of tool names the agent actually invoked during that run, which you will extract from the agent’s response object in the next section.

The LLM-as-a-judge check

Rule-based checks cannot evaluate tone, completeness, or whether an answer actually addresses the question asked, only whether a specific string shows up. That gap is what the judge is for:

def llm_judge(user_input, answer, rubric):
    prompt = (
        f"User asked: {user_input}n"
        f"Agent answered: {answer}n"
        f"Rubric: {rubric}nn"
        "Does the answer satisfy the rubric? Reply with only YES or NO."
    )
    response = judge.invoke(prompt).content.strip().upper()
    return response.startswith("YES")

The judge is a plain model call, not an agent, since grading an answer does not need tools. Notice the prompt asks for a single-word verdict rather than an explanation; that keeps the response cheap to parse reliably with .startswith("YES") instead of having to interpret free-form reasoning. In this tutorial the judge happens to be the same qwen3.5:4b model as the agent, which keeps the setup to a single downloaded model, but it does not have to be. The “Common mistakes” section below covers why using a different, typically stronger, model as the judge is often the safer choice once the stakes are higher than a tutorial.

Tie it together in run_evals()

Finally, the function that runs every test case through the agent, applies all three checks, and prints a summary:

def run_evals():
    agent = build_agent()
    passed_count = 0
    for i, case in enumerate(TEST_CASES, start=1):
        result = agent.invoke({"messages": [{"role": "user", "content": case["input"]}]})
        answer = result["messages"][-1].content

        tool_calls = []
        for msg in result["messages"]:
            calls = getattr(msg, "tool_calls", None)
            if calls:
                tool_calls.extend(c["name"] for c in calls)

        substring_ok = check_substring(answer, case["expected_substring"])
        tool_ok = check_tool(tool_calls, case["expected_tool"])
        judge_ok = llm_judge(case["input"], answer, case["judge_rubric"])
        passed = substring_ok and tool_ok and judge_ok
        if passed:
            passed_count += 1

        status = "PASS" if passed else "FAIL"
        print(f"[{status}] Test {i}: {case['input']}")
        print(f"    answer: {answer!r}")
        print(f"    tools called: {tool_calls}")
        if not substring_ok:
            print(f"    - substring check failed (expected {case['expected_substring']!r})")
        if not tool_ok:
            print(f"    - tool check failed (expected {case['expected_tool']!r}, got {tool_calls})")
        if not judge_ok:
            print("    - judge said NO")

    print(f"n{passed_count}/{len(TEST_CASES)} passed")


if __name__ == "__main__":
    run_evals()

The loop pulls the agent’s final answer out of result["messages"][-1].content, then walks every message in the conversation looking for a tool_calls attribute to build the list of tools actually used. A test only counts as a pass when all three checks agree; if any one of them fails, the whole case fails and the printed diagnostics tell you exactly which check caught the problem.

Step 5: Run the evals

With Ollama still running in the background from Step 1, run the harness:

python3 eval_agent.py

Expected output: this is the real, complete output this exact script produced while writing this tutorial, on a CPU-only host with no GPU:

[PASS] Test 1: What time is it right now?
    answer: "It's currently **July 18, 2026 at 02:43:42 AM**."
    tools called: ['current_time']
[PASS] Test 2: How many vowels are in the word Kubernetes?
    answer: 'There are **4** vowels in the word "Kubernetes". nnThe vowels are: u, e, e, and e.'
    tools called: ['count_vowels']
[PASS] Test 3: What is the capital of Japan?
    answer: 'The capital of Japan is Tokyo.'
    tools called: []

3/3 passed

The full run took a little over four minutes on that CPU-only host, since each test case makes two sequential model calls (one for the agent, one for the judge) and CPU inference is meaningfully slower than GPU inference. Expect it to run noticeably faster on a machine with a GPU that Ollama can use, and expect it to scale roughly linearly as you add more test cases. Rerun this same command any time you change the system prompt, swap the model, or add a new tool; that repeatability, not any single run, is the actual point of building a harness instead of testing by hand in a chat window.

Step 6: Read a failing check (and why one signal is not enough)

A harness you have never seen fail is a harness you should not fully trust yet. To see a real failure, temporarily add a fourth case to TEST_CASES with a deliberately wrong expectation:

    {
        "input": "How many vowels are in the word Docker?",
        "expected_substring": "2",
        "expected_tool": "current_time",  # deliberately wrong on purpose
        "judge_rubric": "The answer should say the vowel count is 2.",
    },

“Docker” does have exactly two vowels (o and e), so the substring check and the judge should both agree the answer is correct. The expected_tool value, though, is set to current_time on purpose, a tool this question has no reason to call. Rerunning the harness with this case added produced this real output:

[FAIL] Test 4: How many vowels are in the word Docker?
    answer: 'There are **2** vowels in the word "Docker" - specifically the letters 'o' and 'e'.'
    tools called: ['count_vowels']
    - tool check failed (expected 'current_time', got ['count_vowels'])

Look closely at what did not get flagged: the substring check passed (the answer does contain “2”), and the judge check passed too (the answer plainly satisfies “the vowel count is 2”). Only the tool check failed, because it is the only one of the three that looks at how the agent produced its answer rather than just the final text. A judge grading on answer quality alone would have waved this case through as a clean pass. That is precisely the case for keeping rule-based checks around even after you add a judge: a wrong or wasteful tool choice can hide behind a correct-looking answer, and only a check that inspects the agent’s actual behavior will catch it. Remove the test case once you have seen this for yourself, since it does not belong in a suite that is supposed to represent real expected behavior.

Common mistakes and how to catch them

• Skipping temperature=0 on the judge. A judge sampling at a higher temperature can grade the exact same answer differently across two runs, which turns your pass/fail history into noise you cannot trust. Set it explicitly on the judge model, and treat any drift in verdicts on unchanged code as a signal your rubric is ambiguous, not just bad luck.
• Vague rubrics. A rubric like “the answer should be good” gives the judge nothing concrete to check against, and models tend to default to a generous YES when a rubric is fuzzy. Write rubrics as a single, falsifiable claim, the same way you would phrase a unit test assertion: “the answer should say Tokyo,” not “the answer should be about geography.”
• Using the agent’s own model as the only judge in a high-stakes eval. This tutorial uses one model for both roles to keep the setup to a single download, which is fine for learning the pattern. A model grading its own output can share the exact same blind spots as the agent, so once you are evaluating something that actually matters, prefer a separate, typically stronger, model as the judge.
• Forgetting to activate the virtual environment. A fresh terminal tab does not inherit an already-activated venv. If python3 eval_agent.py suddenly reports ModuleNotFoundError: No module named 'langchain' after working a moment ago, run source venv/bin/activate again in that shell before anything else.
• Forgetting to pull the model first. If Ollama is running but the model tag was never pulled, ChatOllama calls fail with a clear “model not found” style error from Ollama’s API rather than a Python traceback deep in LangChain. Run ollama pull qwen3.5:4b (or ollama list to check what is already available) and retry.
• Testing only the happy path. Three test cases that all pass on the first try tell you the harness works, not that the agent is reliable. Add cases you expect to be hard: ambiguous phrasing, a question that could plausibly trigger the wrong tool, an input with no correct tool at all. A suite that never fails anything is rarely testing anything interesting.

Verifying everything works end to end

Before trusting this harness, confirm both directions: that it passes what it should, and that it catches what it should not.

1. Run python3 eval_agent.py with the three original test cases in place and confirm the final line reads 3/3 passed, matching the real run captured in Step 5.
2. Add the deliberately wrong test case from Step 6, rerun, and confirm you see exactly one [FAIL] line with a tool check failed message and a final line reading 3/4 passed.
3. Remove that deliberate test case again, so the suite goes back to representing real expected behavior rather than a planted failure.
4. Make one real change, for example editing the current_time tool’s docstring to something misleading, and rerun. If the harness is doing its job, a change that makes the agent behave worse should show up as a new failure, not silently pass.

Every command, code block, and output shown in this tutorial, including both the passing run in Step 5 and the deliberate failure in Step 6, was executed against a live, locally running qwen3.5:4b model while writing this tutorial, using the exact agent.py and eval_agent.py shown above. Nothing here is a guess at what the output “should” look like.

Next steps

• Grow the test suite past a handful of cases, and organize it around the specific ways your real agent has failed in the past, not just generic examples.
• Wire eval_agent.py into a CI step or a pre-commit hook, so a prompt or model change that regresses the agent cannot merge without at least a visible warning.
• Swap in a separate, stronger model as the judge only, keeping a smaller or cheaper model as the agent under test, to reduce the risk of the judge and the agent sharing the same blind spots.
• Log each run’s pass count and timestamp to a file instead of only printing it, so you can track a slow decline across weeks, not just catch a hard failure the moment it happens.

Sources: freeCodeCamp: How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python (Darsh Shah); LangChain documentation: Agents overview and create_agent reference; Ollama: Download Ollama on Linux; Ollama Library: qwen3.5. Featured image: courtroom gavel photograph by Joe Gratz, CC0 1.0 Universal (public domain), via Wikimedia Commons.