How to Automatically Tune RAG Chunk Size and Top-K With a Local Eval Harness

If you have ever built a retrieval-augmented generation (RAG) pipeline, you have hit this moment: your documents are chunked, your embeddings are indexed, your questions get answered, and someone asks “why did we pick a chunk size of 500? Why top 3 results and not top 5?” The honest answer is usually “a blog post used that number and it seemed fine.” That answer does not hold up once the pipeline is answering real questions for real people.

This tutorial builds an automated evaluation harness that replaces the guess with a measurement. You will write a small internal knowledge base, a golden set of questions with known correct facts, and a Python script that sweeps a grid of chunk sizes and retrieval counts (called top_k), scores each combination on whether it actually retrieves and states the right facts, and locks in the winner. Everything runs locally with Ollama, so you can follow along without an API key or a cloud bill.

Along the way you will see real failures: a chunk boundary that splits a command in half, a small model that confidently states the wrong number because it borrowed a fact from the wrong section, and an LLM judge that approves an answer a human would immediately reject. These are not staged examples. They are the actual output from running this code.

What you will build, in plain language

Retrieval-augmented generation means: instead of asking a language model to answer a question from memory, you first search your own documents for the passages most likely to contain the answer, then hand those passages to the model as context and ask it to answer using only that context. This is how you get a chatbot that can accurately answer questions about your company’s internal policies instead of hallucinating plausible-sounding nonsense.

Two settings control how well this works, and neither has a universal right answer:

• Chunk size. Before you can search your documents, you have to cut them into smaller pieces (chunks), because embedding models and language models both have limited context windows and work better on focused passages than on entire documents. Cut chunks too small and a single fact can get split across a chunk boundary, so no single chunk contains the whole answer. Cut chunks too large and you waste retrieval slots on irrelevant text, and the language model has to work harder to find the one sentence that matters.
• Top-k. This is how many chunks you retrieve and hand to the model for each question. A small top_k (like 1) is fast and focused but fragile: if the single best-matching chunk happens to be missing the fact you need, you get nothing. A larger top_k gives the model more chances to see the right fact, at the cost of more irrelevant text mixed in.

The right values depend on your specific documents and the kinds of questions people ask, which is exactly why you cannot copy a number from a tutorial and expect it to hold. What you can copy is the harness that measures it for you.

Prerequisites

• Python 3.10 or later. This tutorial was built and tested on Python 3.13.14.
• Ollama installed and running locally. This tutorial uses Ollama 0.32.9. If you have not installed Ollama before, see our guide to installing Ollama and running your first local chatbot first.
• Two Ollama models pulled: an embedding model and a small chat model:

  ollama pull nomic-embed-text
  ollama pull qwen2.5:1.5b

  Any small instruction-tuned chat model will work; qwen2.5:1.5b is used here because it is fast enough to run a full sweep in well under a minute on this machine’s 4GB GPU.

• The requests and pytest Python packages:

  pip install requests pytest

• Comfort with basic Python (functions, dictionaries, list comprehensions) and running commands in a terminal. No prior RAG or vector database experience is assumed; every concept is explained as it comes up.

Step 1: Build a small internal knowledge base

You need documents worth searching. To keep this reproducible and free of any real company’s data, build a small set of internal engineering runbooks for a fictional company’s on-call team. This is realistic: runbooks are exactly the kind of internal knowledge base a RAG assistant is often built for, and they are dense with the specific numbers and commands that make chunk size actually matter.

Create a docs/ folder with six short text files. Here is one of them in full, so you can see the level of detail; the others follow the same pattern (a rollback procedure, a certificate rotation policy, an on-call escalation policy, a secrets rotation policy, and an incident severity classification document):

docs/db_failover.txt

Database Failover Runbook

Scope: This runbook covers the primary PostgreSQL cluster (orders-db) managed by Patroni,
consisting of one primary node and two synchronous replicas in the us-east-1 region.

When to trigger a failover

Trigger an automatic failover if either of these conditions holds for longer than the
stated window:

- Replication lag on the primary exceeds 30 seconds, measured by the lag_seconds metric
  in the patroni_status dashboard.
- The primary node fails health checks and is unreachable for more than 45 seconds,
  as reported by the Patroni REST API on port 8008.

How to perform a failover

1. Confirm the replica you intend to promote is caught up by checking that its
   replication lag is under 5 seconds.
2. Run the failover from the orchestration host:

   patronictl -c /etc/patroni/patroni.yml failover --candidate replica-2

3. Patroni will pause writes, promote replica-2 to primary, and reconfigure the
   remaining replica to stream from the new primary.
...

Notice that the two most important numbers in this document, the 30-second replication lag threshold and the exact patronictl command, sit close together but are still separated by several sentences. That gap is exactly what will trip up small chunk sizes later.

The other five documents (cert_rotation.txt, rollback_procedure.txt, oncall_escalation.txt, secrets_rotation.txt, and incident_severity.txt) each define a handful of specific thresholds and commands the same way: an error-rate rollback trigger, certificate rotation windows, acknowledgment SLAs by severity, credential rotation cadences, and postmortem deadlines. Each document runs about 1,600 to 1,930 characters once whitespace is normalized, roughly the length of a real internal runbook section.

Step 2: Build a golden evaluation set

A golden set is a list of questions paired with the fact that a correct answer must contain, plus which document that fact lives in. This is the ruler you will measure every configuration against. Without it, “does this chunk size work well” is just a feeling.

golden_set.json

[
  {
    "id": "q1",
    "question": "How many seconds of replication lag on the primary should trigger a database failover?",
    "source_doc": "db_failover.txt",
    "key_fact": "30 seconds"
  },
  {
    "id": "q2",
    "question": "What command should I run to promote replica-2 during a Patroni failover?",
    "source_doc": "db_failover.txt",
    "key_fact": "patronictl -c /etc/patroni/patroni.yml failover"
  },
  {
    "id": "q6",
    "question": "Within how many minutes must a SEV1 incident be acknowledged by on-call?",
    "source_doc": "oncall_escalation.txt",
    "key_fact": "15 minutes"
  }
]

The full set used in this tutorial has 10 questions spread across all six documents. Each key_fact is a short, specific string, a number with its unit, or an exact command, chosen deliberately so that “does the retrieved text or the generated answer contain this string” is a fair, checkable test rather than something that needs human judgment. Writing your own golden set for a real project takes real effort (expect an hour or more for a modest knowledge base), but it is effort you only pay once, and it is what turns “I think this works” into “I measured that this works.”

Step 3: Write a configurable chunker

The chunker needs to accept a target chunk size and an overlap (a small amount of shared text between consecutive chunks, so a fact sitting right on a boundary has a second chance to appear whole in the next chunk). It should also avoid cutting words in half where it reasonably can.

chunker.py

def chunk_text(text, chunk_size, overlap):
    text = " ".join(text.split())  # normalize whitespace
    if overlap >= chunk_size:
        raise ValueError("overlap must be smaller than chunk_size")

    chunks = []
    start = 0
    n = len(text)
    while start < n:
        end = min(start + chunk_size, n)
        if end < n:
            # snap forward to the next space so we don't cut a word in half
            snap = text.rfind(" ", start, end)
            if snap > start:
                end = snap
        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        if end >= n:
            break
        start = end - overlap
        if start < 0:
            start = 0
    return chunks


def chunk_document(doc_id, text, chunk_size, overlap):
    pieces = chunk_text(text, chunk_size, overlap)
    return [
        {"doc_id": doc_id, "chunk_index": i, "text": piece}
        for i, piece in enumerate(pieces)
    ]

The word-snapping logic matters less than you might expect, as you will see in a moment: even with clean word boundaries, a chunk can still stop in the middle of the one sentence you needed.

What "chunk size" actually looks like

Run the chunker on a small sample to build intuition before using it for real:

>>> from chunker import chunk_text
>>> sample = "one two three four five six seven eight nine ten " * 5
>>> for size in (200, 500, 1200):
...     overlap = max(1, size // 6)
...     cks = chunk_text(sample, size, overlap)
...     print(f"chunk_size={size} overlap={overlap} -> {len(cks)} chunks, lengths={[len(c) for c in cks]}")
...
chunk_size=200 overlap=33 -> 2 chunks, lengths=[199, 78]
chunk_size=500 overlap=83 -> 1 chunks, lengths=[244]
chunk_size=1200 overlap=200 -> 1 chunks, lengths=[244]

A larger chunk size than the input text just returns the whole thing as one chunk, which is exactly what you want: no artificial splitting when there is nothing to split.

Step 4: Embed text with Ollama, and a real gotcha with prefixes

To search chunks by meaning rather than exact keyword match, you need embeddings: numeric vectors where semantically similar text ends up close together. Ollama exposes this through its /api/embed endpoint.

embed_client.py

import math
import requests

OLLAMA_URL = "http://127.0.0.1:11434"
EMBED_MODEL = "nomic-embed-text"


def embed(texts, task_prefix=None):
    if isinstance(texts, str):
        texts = [texts]
    if task_prefix:
        payload_input = [f"{task_prefix}: {t}" for t in texts]
    else:
        payload_input = list(texts)
    resp = requests.post(
        f"{OLLAMA_URL}/api/embed",
        json={"model": EMBED_MODEL, "input": payload_input},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["embeddings"]


def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)

Cosine similarity is just a way of measuring the angle between two vectors: 1.0 means they point in exactly the same direction (as similar as possible), 0.0 means they are unrelated, and it ignores vector length, so a short chunk and a long chunk can still be compared fairly.

Here is the gotcha. Nomic AI's own model card for nomic-embed-text states plainly: "the text prompt must include a task instruction prefix, instructing the model which task is being performed." The two prefixes that matter for retrieval are search_document (for the text you are indexing) and search_query (for the question you are searching with). Skip them, and the model was not trained to treat your input the way you are using it.

Rather than take that on faith, test it. Index the knowledge base with correctly prefixed chunks, then compare three conditions for the same 10 golden questions: correct asymmetric prefixes, the same prefix on both sides (a common copy-paste mistake), and no prefix at all. The metric here is the similarity margin: how much higher the best-matching chunk from the correct document scores than the best-matching chunk from any wrong document, averaged across all 10 questions.

--- Condition A: correct asymmetric prefixes ---
correct prefixes: avg margin (correct-best minus wrong-best) = +0.1494, avg rank of first correct chunk = 1.00

--- Condition C: same prefix on both sides (search_document/search_document) ---
mismatched prefixes: avg margin (correct-best minus wrong-best) = +0.1340, avg rank of first correct chunk = 1.00

--- Condition B: no prefix on either side ---
no prefixes: avg margin (correct-best minus wrong-best) = +0.1567, avg rank of first correct chunk = 1.00

This is worth being honest about: on this particular knowledge base, the prefix choice did not flip which document ranked first, and the margin did not even consistently favor the documented convention. That is a real result, not a rounding error, and it teaches something useful on its own: this six-document knowledge base is topically easy (a database runbook and a secrets policy do not use much overlapping vocabulary), so a coarse "which document" test is not sensitive enough to detect a prefix effect that a harder, more ambiguous corpus would likely reveal. The lesson is not "prefixes don't matter," it is "your evaluation set can be too easy to catch a real problem, and passing an easy test is not the same as confirming a documented model convention is safe to ignore." Follow Nomic's documented convention regardless: it is what the model was trained on, and the rest of this tutorial uses it throughout.

Step 5: Stage 1, a cheap retrieval-only sweep

Now sweep a grid of configurations and measure retrieval quality before spending any time on generation, which is the expensive part. The grid here is three chunk sizes (150, 400, and 900 characters) times two top_k values (1 and 3), six configurations total.

The metric matters more than it first appears. A naive test would check "did the correct document show up in the results." That test is too forgiving: a chunk can come from the exactly right document and still miss the one sentence that has the number you need, if that sentence landed in a different chunk. So the real check is whether the retrieved chunk text itself contains the key fact:

stage1_retrieval_sweep.py (evaluation function)

def _normalize(s):
    return " ".join(s.lower().replace("%", " percent").split())


def evaluate_config(golden, chunks, vecs, top_k):
    hits = 0
    miss_details = []
    for item in golden:
        q_vec = embed(item["question"], task_prefix="search_query")[0]
        scored = sorted(
            zip(chunks, vecs),
            key=lambda cv: cosine_similarity(q_vec, cv[1]),
            reverse=True,
        )
        top = scored[:top_k]
        needle = _normalize(item["key_fact"])
        found = any(needle in _normalize(c["text"]) for c, _ in top)
        if found:
            hits += 1
        else:
            # record the question, expected fact, and what actually came
            # back, so a failure is debuggable instead of just a number
            miss_details.append({"question": item["question"], "key_fact": item["key_fact"]})
    return hits / len(golden), miss_details

Running the full grid against real embeddings takes under four seconds, since there is no generation involved yet:

chunk_size  top_k  n_chunks  hit_rate
----------------------------------------
       150      1        87      60%
       150      3        87      90%
       400      1        33      80%
       400      3        33     100%
       900      1        17     100%
       900      3        17     100%

--- Leaderboard (best hit_rate first) ---
1. chunk_size= 900 top_k=1 -> hit_rate=100% (17 chunks)
2. chunk_size= 900 top_k=3 -> hit_rate=100% (17 chunks)
3. chunk_size= 400 top_k=3 -> hit_rate=100% (33 chunks)
4. chunk_size= 150 top_k=3 -> hit_rate=90% (87 chunks)
5. chunk_size= 400 top_k=1 -> hit_rate=80% (33 chunks)
6. chunk_size= 150 top_k=1 -> hit_rate=60% (87 chunks)

The 150-character, top_k=1 configuration misses 4 of 10 questions. Look at what it actually retrieved for two of those misses, straight from the real run:

Q: What command should I run to promote replica-2 during a Patroni failover?
   expected key_fact: 'patronictl -c /etc/patroni/patroni.yml failover' (from db_failover.txt)
   retrieved [db_failover.txt]: 'l failover --candidate replica-2 3. Patroni will pause writes, promote replica-2 to primary, and reconfigure the'

Q: How often must database passwords be rotated under the standard cadence?
   expected key_fact: '30 days' (from secrets_rotation.txt)
   retrieved [secrets_rotation.txt]: '...Standard rotation cadence - Database passwords must be rotated every 30'

Look closely at both. The first chunk literally starts mid-word: "l failover --candidate replica-2". The chunk boundary landed right inside patronictl, chopping off everything before "l". The second chunk ends with "...every 30", one word short of the unit that makes the number meaningful. This is not a hypothetical failure mode you are told to imagine. It is the literal, reproducible output of running a 150-character chunker against real text.

Step 6: Stage 2, generation and faithfulness

Retrieval hit-rate tells you whether the right fact was available to the model. It does not tell you whether the model actually used it correctly. For that you need to run generation: feed the retrieved chunks to a language model as context, ask it to answer, and check the answer. Because this stage costs real inference time, run it only for a shortlist of configurations rather than the whole grid: the two extremes at top_k=1 (150 and 900 characters), the middle ground (400 characters, top_k=1), and 150 characters with top_k=3, to see whether a larger top_k can compensate for small chunks.

Score each answer two ways. First, a deterministic grounding check: does the generated answer contain the exact key fact, the same substring match used in Stage 1. Second, an LLM-judge call: ask a separate prompt to grade whether the answer correctly states the required fact, replying only YES or NO.

stage2_generation_sweep.py (generation and judging)

def generate_answer(question, context_chunks):
    context = "\n\n".join(c["text"] for c in context_chunks)
    resp = requests.post(
        f"{OLLAMA_URL}/api/chat",
        json={
            "model": GEN_MODEL,
            "messages": [
                {"role": "system", "content": (
                    "Answer the question using ONLY the provided context. "
                    "If the context does not contain the answer, say you "
                    "do not know. Be concise: one or two sentences."
                )},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
            ],
            "stream": False,
            "options": {"temperature": 0, "seed": 42},
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json()["message"]["content"].strip()


def judge_answer(question, key_fact, answer):
    resp = requests.post(
        f"{OLLAMA_URL}/api/chat",
        json={
            "model": GEN_MODEL,
            "messages": [
                {"role": "system", "content": (
                    "You are grading whether an answer correctly states a "
                    "specific required fact. Reply with exactly one word: "
                    "YES if the answer states the required fact correctly, "
                    "NO if it is missing, wrong, or says it doesn't know."
                )},
                {"role": "user", "content": (
                    f"Question: {question}\nRequired fact: {key_fact}\n"
                    f"Answer to grade: {answer}\n"
                    "Does the answer correctly state the required fact? Reply YES or NO only."
                )},
            ],
            "stream": False,
            "options": {"temperature": 0, "seed": 42},
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json()["message"]["content"].strip().upper().startswith("Y")

Running the full shortlist, 4 configurations times 10 questions, 40 generation calls plus 40 judge calls, took 35 seconds on this machine once the model was warm:

chunk_size= 150 top_k=1 -> grounded=70% judge=80% (8.7s for 10 questions)
chunk_size= 400 top_k=1 -> grounded=80% judge=90% (8.8s for 10 questions)
chunk_size= 900 top_k=1 -> grounded=100% judge=100% (8.6s for 10 questions)
chunk_size= 150 top_k=3 -> grounded=80% judge=90% (8.7s for 10 questions)

Two things stand out. The 900-character chunks fully recovered to 100 percent once they had complete facts to work with, which is reassuring: retrieval quality directly predicted generation quality here. And in every configuration, the judge score is higher than the grounded score, which is the opposite of reassuring. Look at why.

The LLM judge approved answers that were wrong

Every disagreement between the deterministic check and the LLM judge across the entire shortlist ran in the same direction: the judge said YES when the deterministic check said NO. Here are all three, verbatim:

chunk_size=150, top_k=1:
  Q: What command should I run to promote replica-2 during a Patroni failover?
  key_fact: 'patronictl -c /etc/patroni/patroni.yml failover'
  answer: 'You should run the `patroni promote` command to promote replica-2 during a Patroni failover.'
  grounded=False  judge=True

chunk_size=400, top_k=1:
  Q: What command should I run to promote replica-2 during a Patroni failover?
  key_fact: 'patronictl -c /etc/patroni/patroni.yml failover'
  answer: 'patronictl -c /etc/patroni/patroni.yml promote 2'
  grounded=False  judge=True

chunk_size=150, top_k=3:
  Q: What sustained error rate triggers an automatic deployment rollback?
  key_fact: '2 percent'
  answer: 'An automatic rollback is triggered when a standard canary rollout pipeline sustains a certain error rate.'
  grounded=False  judge=True

patroni promote is not a real command in this runbook; the model invented a plausible-sounding one after being handed a truncated chunk. patronictl -c /etc/patroni/patroni.yml promote 2 is closer but still wrong, since the actual subcommand is failover --candidate replica-2, not promote 2. And the third answer does not state a number at all, it just restates the question in different words. A human reviewer would reject all three immediately. The LLM judge, asked directly and explicitly to check for the required fact, approved all three anyway.

This is the reason the final leaderboard in this tutorial ranks configurations by the deterministic grounded score, not the judge score. An LLM judge can be a useful secondary signal for softer qualities like tone or completeness, but on a question with one specific, checkable fact, treat it as optimistic rather than reliable. Keep a deterministic check as your source of truth whenever the correct answer can be written down as an exact string.

A confidently wrong answer that both checks caught

Not every failure was subtle. One answer under the 150-character, top_k=1 configuration swapped in the wrong severity level's number entirely:

Q: Within how many minutes must a SEV1 incident be acknowledged by on-call?
key_fact: '15 minutes'
answer: 'Within 30 minutes, a SEV1 incident must be acknowledged by on-call.'
grounded=False  judge=False

The source document lists SEV1 at 15 minutes and SEV2 at 30 minutes, one line apart. At 150 characters, the retrieved chunk happened to contain the SEV2 and SEV3 lines but not the SEV1 line, so the model picked the nearest plausible number in its context and stated it with full confidence, formatted exactly like a correct answer. This is arguably more dangerous than an obvious "I don't know," because nothing about the answer's tone signals that it is wrong. Both the deterministic check and the judge caught this one, which is reassuring, but it is a preview of exactly the kind of error a production RAG assistant can ship silently if nobody is measuring it.

A case where generation succeeded despite retrieval failing

One more result is worth pulling out because it cuts the other way. The database password rotation question was a confirmed Stage 1 miss at 150 characters (the retrieved chunk ended at "...every 30", missing the word "days"), yet Stage 2 generation for the same configuration produced a fully correct answer: "Database passwords must be rotated every 30 days under the standard cadence."

The model filled in "days" from its own general knowledge; 30-day rotation is a common enough default that a reasonable guess happened to land on the right unit. That is not a win to bank on. If you only measured end-to-end generation accuracy, this case would look like a pass and hide the fact that retrieval was genuinely broken for this question. A different fact, one with a less guessable unit or value, would not get the same lucky save. This is exactly why this tutorial tracks retrieval-level grounding and generation-level grounding as two separate numbers instead of collapsing them into one: a generation success paired with a retrieval failure is a flag worth a manual look, not proof that the configuration is safe.

Step 7: Build the combined leaderboard and lock in a winner

Combine both stages into one table, rank by the deterministic grounded rate, and save the winner as a small JSON file your application can load at startup instead of hardcoding numbers:

final_leaderboard.py (ranking and save)

ranked = sorted(rows, key=lambda r: (-r["grounded_rate"], -r["retrieval_hit_rate"]))
winner = ranked[0]

with open("winning_config.json", "w") as f:
    json.dump({
        "chunk_size": winner["chunk_size"],
        "top_k": winner["top_k"],
        "overlap_ratio": 0.15,
        "embed_model": "nomic-embed-text",
        "gen_model": "qwen2.5:1.5b",
        "selected_from_grounded_rate": winner["grounded_rate"],
    }, f, indent=2)

chunk_size  top_k  retrieval  grounded   judge
------------------------------------------------
       150      1        60%       70%     80%
       400      1        80%       80%     90%
       900      1       100%      100%    100%
       150      3        90%       80%     90%

Winner (ranked by deterministic grounded_rate, not judge_rate):
  chunk_size=900 top_k=1 grounded=100% retrieval=100%

Saved winning_config.json

Notice the 900-character, top_k=1 configuration wins cleanly here, and it also happens to be the cheapest to run: the fewest chunks to embed (17, versus 87 at 150 characters) and only one chunk retrieved per question. Bigger is not always more expensive in a RAG pipeline; sometimes the config that needs the least infrastructure is also the one that performs best, because fewer, more complete chunks give the retriever less room to make a partial match.

Step 8: Verify the winning configuration end to end

A configuration that wins on your 10 golden questions still needs to work on a question you did not train it on. Load the saved config and try a brand-new question that was never part of the evaluation set:

verify_end_to_end.py

with open("winning_config.json") as f:
    cfg = json.load(f)

chunks, vecs = build_index(cfg["chunk_size"])

new_question = (
    "What is the most common mistake that causes outages during certificate "
    "rotation, and how long should I wait before removing the old certificate?"
)
top = retrieve(new_question, chunks, vecs, cfg["top_k"])
answer = generate_answer(new_question, top)

Loaded winning config: chunk_size=900 top_k=1

Retrieved chunk(s):
  [cert_rotation.txt] "hich skips the canary step and pages the security on-call engineer as
  part of the process. Common mistakes Removing the old certificate from the
  load balancer's store before the new one has been verified on all hosts is
  the most common cause of rotation-related outages, since some hosts may
  still be serving stale configuration from a cached reload."

Question: What is the most common mistake that causes outages during certificate rotation, and how long should I wait before removing the old certificate?
Answer:   The most common mistake causing outages during certificate rotation is
removing the old certificate from the load balancer's store before verifying
it on all hosts. You should wait until all hosts have been updated with the
new configuration to avoid stale configuration issues.

The first half of the answer is correct and well-grounded. The second half is accurate in spirit but incomplete: the source document actually specifies "24 hours of clean traffic" as the concrete wait time, and that number lives in a different 900-character chunk than the "common mistakes" section that got retrieved. With top_k=1, only one chunk comes back, so a question that genuinely needs facts from two different places in the same document will only get the one that scored higher. This is a real, checkable limitation of the winning configuration, and it is worth knowing about even though it does not show up in the 10-question golden set. In production, if you expect multi-part questions to be common, that is a good reason to accept a small efficiency cost and bump top_k to 2 or 3, even though the narrower golden set scored top_k=1 as the winner.

Common mistakes and gotchas

• Testing only document-level retrieval. "Did the right document come back" is a much easier test to pass than "did the right chunk contain the right fact," and passing the easy version can hide a real problem. Every configuration in this tutorial hit 100 percent on a document-level test; only the fact-level test revealed the 150-character configuration's real 60 percent failure rate.
• Trusting an LLM judge as your only faithfulness signal. The judge in this tutorial approved three answers a deterministic check correctly rejected, including two different fabricated variants of the same command. Use a deterministic, string-based check for any fact that can be written down exactly, and treat an LLM judge as a supplementary signal at best.
• Skipping the documented embedding prefixes because a quick test did not show damage. This tutorial's own prefix test came back inconclusive on an easy six-document corpus. That is a property of the test, not permission to skip a documented model convention. Follow the model card.
• Judging a configuration only on end-to-end generation accuracy. The database password rotation case showed a model producing a correct answer despite a genuinely broken retrieval step, by guessing a common default. Track retrieval-level and generation-level scores separately so a lucky guess does not get credited as a working pipeline.
• Assuming the golden set's winner covers every real question. The certificate rotation follow-up question showed the "winning" configuration giving an accurate but incomplete answer to a multi-part question outside the evaluation set. A small golden set (10 questions here) narrows your search space; it does not replace watching what the system does on the long tail of real user questions.

How to confirm the whole thing works, and what to try next

To confirm your own version of this harness is working correctly end to end, check three things: the unit tests for the chunker and scoring logic pass without needing Ollama running at all, the Stage 1 sweep produces a leaderboard with real variation between configurations (if every row shows 100 percent, your golden set is too easy and needs harder or more ambiguous questions), and the Stage 2 sweep's grounded scores are lower than or equal to its judge scores in most disagreements, which is the signature of a judge that is too generous rather than too strict.

$ python -m pytest test_harness.py -v
...
test_harness.py::test_chunks_respect_size_budget PASSED
test_harness.py::test_chunks_cover_the_whole_text_with_overlap PASSED
test_harness.py::test_chunk_document_tags_doc_id_and_index PASSED
test_harness.py::test_cosine_similarity_identical_vector_is_one PASSED
test_harness.py::test_cosine_similarity_orthogonal_is_zero PASSED
test_harness.py::test_normalize_converts_percent_sign PASSED
test_harness.py::test_normalize_is_case_and_whitespace_insensitive PASSED
test_harness.py::test_overlap_must_be_smaller_than_chunk_size PASSED

8 passed in 0.09s

From here, a few natural directions to extend this harness:

• Add an overlap sweep alongside chunk size and top_k; this tutorial fixed overlap at 15 percent of chunk size to keep the grid small, but overlap has its own tradeoffs worth measuring.
• Swap in a second embedding model and add it as a third grid dimension, so "which embedding model" gets the same rigorous treatment as chunk size and top_k instead of being chosen by reputation.
• If your real documents are longer and more numerous than this tutorial's six short runbooks, read our guide to evaluating RAG retrieval quality with precision, recall, and MRR for a deeper set of retrieval metrics once hit-rate alone stops being discriminating enough.
• If you have not built a full RAG pipeline before this tutorial's stripped-down version, our guide to building a local RAG Q&A agent with LangChain and Ollama covers the end-to-end application, including conversation memory, that this tutorial intentionally left out to keep the focus on evaluation.
• The uncertainty-detection techniques in our guide to building an AI system that knows when it doesn't know pair naturally with this tutorial's grounding checks, since both are about catching a confident wrong answer before a user sees it.