How to Build an AI System That Knows When It Doesn’t Know With Ollama and Python

Large language models are trained to always produce the most statistically likely next word, not to know when they are guessing. Ask a chatbot something it has no real information about, and it will usually still answer, confidently and fluently, even when the answer is invented. In a personal chat that is annoying. In a production system, a business assistant that confidently invents a refund policy or a support bot that confidently invents a printer model number is a real liability.

In this tutorial you will build a small, local, three-layer defense system that catches overconfident AI answers before they reach a user: one layer decides whether a question is even something your assistant should try to answer, a second layer decides whether the information it found is actually relevant, and a third layer inspects the model’s own token-by-token confidence after it answers. You will run every layer against a real local model with Ollama, and you will see, with real captured numbers, both where this approach works well and where it genuinely falls short. This is a beginner-friendly walkthrough: no machine learning background is assumed, and every concept (embeddings, cosine similarity, logprobs, perplexity) is explained before it is used.

What You’ll Build and Why It Matters

You will build a small internal IT/HR helpdesk assistant for a fictional company, Aurora Corp, that answers questions from a handful of real policy documents (VPN setup, password resets, expense limits, PTO, and equipment requests). Around that assistant you will build a decision pipeline with three checkpoints:

• Layer 1, boundary detection: is this question even inside the topics the assistant is supposed to cover?
• Layer 2, retrieval confidence: did we find a policy document that actually matches the question, and is that match unambiguous?
• Layer 3, output confidence: after the model answers, how “sure” was it, token by token, and what does that number actually tell you?

By the end, the assistant will correctly answer well-grounded questions, correctly reject a question about the weather, correctly reject a question your knowledge base cannot answer, and honestly refuse to invent a detail your policy documents never mention. Along the way you will hit a real calibration bug (a correct answer that looked less confident than a fabricated one) and a real, documented gap between what Ollama’s own documentation says about logprobs and what the server actually returns. Both are covered honestly, not glossed over.

Prerequisites

• Windows, macOS, or Linux. This tutorial was built and tested on Windows.
• Python 3.10 or newer (this tutorial used 3.13.14).
• Ollama installed and running locally. If you don’t have it yet, sxz.io’s own guide to installing Ollama and running your first local chatbot covers that from scratch.
• Two Ollama models pulled locally: a small chat model (this tutorial uses qwen2.5:1.5b) and an embedding model (nomic-embed-text). Pull them with ollama pull qwen2.5:1.5b and ollama pull nomic-embed-text if you don’t already have them.
• Comfort writing and running basic Python functions. No prior exposure to embeddings, vector similarity, or token probabilities is assumed; every term is defined the first time it’s used.

The Problem: Why LLMs Can’t Say “I Don’t Know” on Their Own

A language model is trained on one objective: given the text so far, predict the next token. Nothing in that objective rewards the model for pausing and admitting uncertainty. When you ask a well-covered question, the model’s training data gives it a strong, sharp signal about what the next words should be, and the answer tends to be accurate. When you ask something obscure, out of scope, or simply not in the model’s training data, the model still has to pick a next token, so it produces the most plausible-sounding continuation it can, whether or not that continuation is true. This is the core mechanic behind what the industry calls “hallucination”: not a bug in one specific answer, but a predictable consequence of how the model was trained to behave.

The fix is not to make the model magically self-aware. The fix is to build deterministic code around the model that decides, independently of what the model itself claims, whether an answer should be trusted, escalated to a human, or rejected outright. That is what the three layers in this tutorial do.

Step 1: Set Up Your Environment

Install the Python Packages

Create a project folder and an isolated virtual environment, then install the two packages this tutorial needs: requests to talk to Ollama’s HTTP API, and numpy for the vector math behind similarity scoring.

mkdir uncertainty-pipeline
cd uncertainty-pipeline
python -m venv venv
venv\Scripts\activate
pip install requests numpy pytest

This tutorial used requests 2.34.2, numpy 2.5.2, and pytest 9.1.1.

Confirm Ollama Is Running With the Right Models

Run ollama list and confirm both models are present:

NAME                       ID              SIZE      MODIFIED
nomic-embed-text:latest    0a109f422b47    274 MB    5 days ago
qwen2.5:1.5b               65ec06548149    986 MB    3 weeks ago

If either is missing, pull it with ollama pull nomic-embed-text or ollama pull qwen2.5:1.5b. Ollama needs to be running in the background (it usually starts automatically after installation) and reachable at http://localhost:11434, which is its default address.

Step 2: The Three Layers, at a Glance

Before writing code, it helps to understand what each layer is actually measuring and why they run in this specific order:

1. Layer 1 runs first and costs the least. It never touches your knowledge base or your generation model. It just asks: is this question’s meaning close to the kinds of questions we’re supposed to handle at all?
2. Layer 2 runs only if Layer 1 passes. It searches your actual knowledge base and asks two questions: is the best match good enough on its own, and is it clearly better than the second-best match? (You’ll see below why checking only the first question is a real, demonstrated mistake.)
3. Layer 3 runs only if Layer 2 passes, after generation. It looks at the model’s own per-token probabilities for the answer it just gave. This is the layer with the most nuance, and the one most tutorials oversimplify. You’ll see real data showing exactly where it helps and exactly where it misleads.

The key design principle, confirmed by the end of this tutorial with real numbers, is that Layers 1 and 2 do the reliable work because they run before generation and never depend on the model’s self-reported confidence. Layer 3 is useful, but only as a supporting signal.

Step 3: Layer 1, Reject Questions That Are Out of Scope

What an Embedding Actually Is

An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Ollama’s nomic-embed-text model turns any sentence into a vector of 768 numbers. Two sentences with similar meaning produce vectors that point in a similar direction in that 768-dimensional space, even if they don’t share any of the same words. You measure “similar direction” with cosine similarity: a score from -1 to 1, where 1 means the vectors point in exactly the same direction (very similar meaning), 0 means they’re unrelated, and negative values mean opposite meaning. In practice, for short real-world sentences, you’ll mostly see scores between 0 and 1.

Layer 1 works by comparing a new question’s embedding against a small set of example questions the assistant is supposed to handle, and checking how close the best match is.

Building the Boundary Check

First, a small helper module for talking to Ollama. This is the only file that makes HTTP requests; every other module imports from it.

"""Small helper functions for talking to a local Ollama server."""
import requests

OLLAMA_URL = "http://localhost:11434"


def embed(text: str, model: str = "nomic-embed-text") -> list[float]:
    resp = requests.post(
        f"{OLLAMA_URL}/api/embed",
        json={"model": model, "input": text},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["embeddings"][0]


def chat_with_logprobs(prompt: str, model: str = "qwen2.5:1.5b", max_tokens: int = 60, temperature: float = 0.0):
    resp = requests.post(
        f"{OLLAMA_URL}/v1/chat/completions",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
            "temperature": temperature,
            "logprobs": True,
            "top_logprobs": 1,
        },
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    choice = data["choices"][0]
    text = choice["message"]["content"]
    token_logprobs = [t["logprob"] for t in choice["logprobs"]["content"]]
    return text, token_logprobs


if __name__ == "__main__":
    vec = embed("What is our VPN setup process?")
    print("embedding dims:", len(vec))
    text, lps = chat_with_logprobs("What is 2+2?")
    print("answer:", text)
    print("token logprobs:", [round(x, 3) for x in lps])

Now the boundary check itself. It holds seven example questions that represent the kinds of things the assistant should answer, embeds them once, and compares any new question against all seven, keeping the single best match:

"""Layer 1: is this question even something our assistant is supposed to answer?"""
import numpy as np
from ollama_client import embed

# A handful of real questions the assistant is *supposed* to handle.
IN_DOMAIN_EXAMPLES = [
    "How do I connect to the company VPN?",
    "How do I reset my network password?",
    "What is the maximum amount I can expense without a receipt?",
    "How many paid time off days do I get per year?",
    "How do I request a new laptop or monitor?",
    "Who approves expense reports over $500?",
    "What is the process for reporting a lost company phone?",
]

_in_domain_vectors = None


def _reference_vectors():
    global _in_domain_vectors
    if _in_domain_vectors is None:
        _in_domain_vectors = np.array([embed(q) for q in IN_DOMAIN_EXAMPLES])
    return _in_domain_vectors


def cosine_similarity(a, b) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))


def boundary_check(query: str, threshold: float = 0.55):
    query_vec = embed(query)
    sims = [cosine_similarity(query_vec, ref) for ref in _reference_vectors()]
    best_idx = int(np.argmax(sims))
    best_score = sims[best_idx]
    return {
        "query": query,
        "in_domain": best_score >= threshold,
        "best_score": round(best_score, 4),
        "closest_reference": IN_DOMAIN_EXAMPLES[best_idx],
    }


if __name__ == "__main__":
    test_queries = [
        "How do I set up the company VPN on my laptop?",
        "Can I expense a $30 lunch without a receipt?",
        "What's the weather like in Tokyo right now?",
        "Write me a poem about a cat",
    ]
    for q in test_queries:
        result = boundary_check(q)
        print(result)

Testing Layer 1 With Real Queries

Run python layer1_boundary.py. Here is the real, captured output from this exact code:

{'query': 'How do I set up the company VPN on my laptop?', 'in_domain': True, 'best_score': 0.8472, 'closest_reference': 'How do I connect to the company VPN?'}
{'query': 'Can I expense a $30 lunch without a receipt?', 'in_domain': True, 'best_score': 0.6919, 'closest_reference': 'What is the maximum amount I can expense without a receipt?'}
{'query': "What's the weather like in Tokyo right now?", 'in_domain': False, 'best_score': 0.3549, 'closest_reference': 'How many paid time off days do I get per year?'}
{'query': 'Write me a poem about a cat', 'in_domain': False, 'best_score': 0.4443, 'closest_reference': 'How do I request a new laptop or monitor?'}

The separation is clean: the two in-domain questions score 0.69 and 0.85, while the two out-of-domain questions score 0.35 and 0.44, comfortably on either side of the 0.55 threshold. Notice the third example: the model isn’t matching against exact words like “weather”, it’s matching meaning, and a weather question simply doesn’t resemble any of the seven reference questions closely enough to pass.

Step 4: Layer 2, Don’t Trust a Weak or Ambiguous Retrieval Match

Building a Tiny Policy Knowledge Base

Layer 2 needs something to retrieve from. Here is Aurora Corp’s entire policy knowledge base: five short, self-contained documents.

"""A tiny, fictional internal IT/HR policy knowledge base for a company called Aurora Corp."""

DOCUMENTS = [
    {
        "id": "vpn-setup",
        "text": (
            "VPN Setup Policy: Aurora Corp employees must connect to the corporate network "
            "using the GlobalProtect VPN client. Install it from the internal software portal, "
            "then log in with your company email and password. Multi-factor authentication is "
            "required on first connection. Contact IT if the client fails to install."
        ),
    },
    {
        "id": "password-reset",
        "text": (
            "Password Reset Policy: Passwords expire every 90 days. To reset a forgotten "
            "password, visit the self-service portal at password.aurora-corp.internal and "
            "verify your identity with your registered phone number. Resets outside business "
            "hours require a call to the IT helpdesk."
        ),
    },
    {
        "id": "expense-limits",
        "text": (
            "Expense Reimbursement Policy: Employees may expense up to $25 per meal without a "
            "receipt. Any single expense over $75 requires an itemized receipt and manager "
            "approval. Expenses over $500 require additional approval from a department "
            "director before reimbursement is processed."
        ),
    },
    {
        "id": "pto-policy",
        "text": (
            "Paid Time Off Policy: Full-time employees accrue 15 PTO days per year during "
            "their first two years, increasing to 20 days after two years of tenure. PTO "
            "requests must be submitted at least 5 business days in advance through the HR "
            "portal for approval by a direct manager."
        ),
    },
    {
        "id": "equipment-requests",
        "text": (
            "Equipment Request Policy: New laptops, monitors, and peripherals can be requested "
            "through the IT asset portal. Standard requests are fulfilled within 5 business "
            "days. Employees may request a replacement device once every three years unless "
            "hardware fails and is documented by IT support."
        ),
    },
]

Retrieving With Both an Absolute Score and a Margin

The naive approach to retrieval confidence is: embed the query, compare it against every document, and if the best match scores above some threshold, trust it. The code below started out exactly that way, with only a min_similarity check. Testing it exposed a real problem, described right after the code, which is why the final version below also checks the margin, the gap between the best-matching document’s score and the second-best document’s score.

"""Layer 2: once a question is in-domain, how good is the retrieved context?"""
import numpy as np
from ollama_client import embed
from layer1_boundary import cosine_similarity
from policy_kb import DOCUMENTS

_doc_vectors = None


def _document_vectors():
    global _doc_vectors
    if _doc_vectors is None:
        _doc_vectors = [(doc, embed(doc["text"])) for doc in DOCUMENTS]
    return _doc_vectors


def retrieve(query: str, min_similarity: float = 0.45, min_margin: float = 0.05):
    query_vec = embed(query)
    scored = [
        (doc, cosine_similarity(query_vec, vec))
        for doc, vec in _document_vectors()
    ]
    scored.sort(key=lambda pair: pair[1], reverse=True)
    best_doc, best_score = scored[0]
    runner_up_score = scored[1][1]
    margin = best_score - runner_up_score
    passes_absolute = best_score >= min_similarity
    passes_margin = margin >= min_margin
    return {
        "query": query,
        "retrieval_confident": passes_absolute and passes_margin,
        "best_score": round(best_score, 4),
        "matched_doc_id": best_doc["id"],
        "matched_text": best_doc["text"],
        "runner_up_score": round(runner_up_score, 4),
        "margin": round(margin, 4),
        "passes_absolute_threshold": passes_absolute,
        "passes_margin_threshold": passes_margin,
    }


if __name__ == "__main__":
    test_queries = [
        "How many vacation days do I earn after 3 years?",
        "Can I expense a $600 conference registration fee?",
        "What is our policy on remote pair programming tools?",
    ]
    for q in test_queries:
        result = retrieve(q)
        print(result["query"])
        print(
            "  confident:", result["retrieval_confident"],
            "| score:", result["best_score"],
            "| runner-up:", result["runner_up_score"],
            "| margin:", result["margin"],
        )
        print("  matched doc:", result["matched_doc_id"])
        print()

A Real Retrieval Mistake I Found While Testing

Running python layer2_retrieval.py against three real questions produced this real, captured output:

How many vacation days do I earn after 3 years?
  confident: True | score: 0.6454 | runner-up: 0.5003 | margin: 0.1451
  matched doc: pto-policy

Can I expense a $600 conference registration fee?
  confident: True | score: 0.5209 | runner-up: 0.4564 | margin: 0.0646
  matched doc: expense-limits

What is our policy on remote pair programming tools?
  confident: False | score: 0.5512 | runner-up: 0.5219 | margin: 0.0293
  matched doc: equipment-requests

The third result is the important one. Aurora Corp has no policy at all about pair programming tools. Yet the absolute similarity score (0.5512) cleared the 0.45 threshold, and the naive version of this code would have confidently handed that document to the generation step, risking an answer that sounds authoritative about a topic the company has no actual policy on. What gives it away is the margin: the best match (0.5512, “equipment-requests”) and the runner-up (0.5219, another document) are only 0.0293 apart, meaning the retrieval system itself is not sure which document is actually relevant. The first two questions have much wider margins (0.1451 and 0.0646), and both are genuinely well-matched.

This is not a staged example. It’s exactly what came out of the first version of this code, and it’s the reason the final retrieve() function above checks passes_margin in addition to passes_absolute. A single similarity threshold is not enough; you need to know that the best match beat its closest competitor by a meaningful amount.

Step 5: Layer 3, Read the Model’s Own Token Confidence

What Logprobs and Perplexity Actually Mean

When a language model generates text, it doesn’t just pick a word, it computes a probability for every possible next word and (usually) samples the highest one. A logprob (log-probability) is the natural logarithm of the probability the model assigned to the specific token it actually produced. Logprobs are always zero or negative: a value near 0 (like -0.01) means the model was very confident in that exact token, while a large negative value (like -5) means the model considered that token quite unlikely, even though it still generated it.

Perplexity is a more intuitive way to look at the same signal. It’s calculated as e raised to the power of the negative average logprob across all tokens in an answer. A perplexity of 1.0 means the model was, on average, completely certain about every token. A perplexity of 2.0 means the model was, roughly speaking, choosing between about two equally likely options at each step. Higher perplexity generally means less confident, more “surprised” generation.

Getting Logprobs Out of Ollama

Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions. Requesting "logprobs": true and "top_logprobs": 1 on that endpoint returns a per-token logprob for every token in the response. Here’s the same helper module from Step 3 again, since chat_with_logprobs() is the function that does this:

"""Small helper functions for talking to a local Ollama server."""
import requests

OLLAMA_URL = "http://localhost:11434"


def embed(text: str, model: str = "nomic-embed-text") -> list[float]:
    resp = requests.post(
        f"{OLLAMA_URL}/api/embed",
        json={"model": model, "input": text},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["embeddings"][0]


def chat_with_logprobs(prompt: str, model: str = "qwen2.5:1.5b", max_tokens: int = 60, temperature: float = 0.0):
    resp = requests.post(
        f"{OLLAMA_URL}/v1/chat/completions",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
            "temperature": temperature,
            "logprobs": True,
            "top_logprobs": 1,
        },
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    choice = data["choices"][0]
    text = choice["message"]["content"]
    token_logprobs = [t["logprob"] for t in choice["logprobs"]["content"]]
    return text, token_logprobs


if __name__ == "__main__":
    vec = embed("What is our VPN setup process?")
    print("embedding dims:", len(vec))
    text, lps = chat_with_logprobs("What is 2+2?")
    print("answer:", text)
    print("token logprobs:", [round(x, 3) for x in lps])

A Documentation Gap Worth Knowing About

Before relying on this, it’s worth checking Ollama’s own documentation, because it says something surprising. Ollama’s official OpenAI-compatibility reference (docs/api/openai-compatibility.mdx in the ollama/ollama repository) lists supported features for the /v1/chat/completions endpoint, and as of this writing it explicitly marks Logprobs as unchecked, meaning not officially supported. That doesn’t match what you just saw: this tutorial’s own test calls against Ollama 0.32.9 returned real, populated logprobs.content data with per-token values and top alternatives every time.

Checking the project’s own issue tracker clears this up. There is an open pull request, “docs: mark Logprobs as supported in OpenAI compatibility”, which confirms the documentation is simply out of date rather than the feature being unreliable. There is also real, relevant history worth knowing about: issue #16196, still open at the time of writing, reports that logprobs are not being properly temperature-scaled, and PR #16321, also still open and unmerged, proposes fixing it. Don’t assume that bug is already resolved on whatever version you’re running; check the issue’s current status first. Separately, issue #18163 (still open at the time of writing) reports that simply requesting logprobs can, on very long generations under greedy decoding, change the exact tokens generated, due to floating-point rounding differences introduced by the extra computation; the reporter’s own data showed short generations were unaffected and only long, multi-thousand-token generations diverged. This tutorial only ever generates short answers (4 to 60 tokens), which is the regime the reporter found unaffected, but if you build on this for long-form generation, read that issue first.

The practical takeaway: this technique works and is genuinely useful, but it is running on a code path its own maintainers haven’t finished documenting yet. Verify it against whatever Ollama version you’re running before depending on it in production, and don’t assume perfect bit-for-bit reproducibility, a point this tutorial returns to with its own evidence in Step 6.

Testing Layer 3 With Four Real Questions

Here’s the scoring function, which turns a list of token logprobs into average logprob, perplexity, and the single worst (most negative) token logprob:

"""Layer 3: after the model answers, how confident was it in its own tokens?"""
import math
from ollama_client import chat_with_logprobs


def score_answer(prompt: str, model: str = "qwen2.5:1.5b", max_tokens: int = 60):
    text, token_logprobs = chat_with_logprobs(prompt, model=model, max_tokens=max_tokens)
    avg_logprob = sum(token_logprobs) / len(token_logprobs)
    perplexity = math.exp(-avg_logprob)
    min_logprob = min(token_logprobs)
    return {
        "prompt": prompt,
        "answer": text.strip(),
        "avg_logprob": round(avg_logprob, 4),
        "perplexity": round(perplexity, 3),
        "min_token_logprob": round(min_logprob, 4),
        "token_count": len(token_logprobs),
    }


if __name__ == "__main__":
    probes = [
        "What is the capital of France? Answer in one short sentence.",
        "What model number is the printer in the third-floor break room at Aurora Corp? Answer in one short sentence.",
        "What is 47 times 36? Answer with just the number.",
        "In one made-up but confident-sounding sentence, tell me the name of Aurora Corp's Chief Astronaut.",
    ]
    for p in probes:
        result = score_answer(p)
        print(f"Q: {result['prompt']}")
        print(f"A: {result['answer']}")
        print(f"   avg_logprob={result['avg_logprob']}  perplexity={result['perplexity']}  min_token_logprob={result['min_token_logprob']}  tokens={result['token_count']}")
        print()

Running python layer3_logprobs.py against four real prompts, none of them grounded in any policy document yet, gives this real, captured output:

Q: What is the capital of France? Answer in one short sentence.
A: The capital of France is Paris.
   avg_logprob=-0.0038  perplexity=1.004  min_token_logprob=-0.0256  tokens=7

Q: What model number is the printer in the third-floor break room at Aurora Corp? Answer in one short sentence.
A: The printer in the third-floor break room at Aurora Corp is model HP LaserJet Pro M423dw.
   avg_logprob=-0.3875  perplexity=1.473  min_token_logprob=-2.2054  tokens=23

Q: What is 47 times 36? Answer with just the number.
A: 1656
   avg_logprob=-0.125  perplexity=1.133  min_token_logprob=-0.3384  tokens=4

Q: In one made-up but confident-sounding sentence, tell me the name of Aurora Corp's Chief Astronaut.
A: "According to our latest press release, Aurora Corp's Chief Astronaut is Dr. Evelyn Vega."
   avg_logprob=-0.504  perplexity=1.655  min_token_logprob=-2.2876  tokens=20

Read these four results carefully, because each one teaches something different:

• “What is the capital of France?” The model answers correctly with a perplexity of 1.004, about as confident as a model can be. This is the baseline: a fact the model has seen thousands of times.
• The printer question is a trap: Aurora Corp’s break room printer model is not something any model could possibly know, since it’s a detail this tutorial invented. The model doesn’t say so. It confidently states “HP LaserJet Pro M423dw”, a completely fabricated model number, with a perplexity of 1.473, noticeably higher than the France answer, but still a plausible-sounding, fluent sentence.
• “What is 47 times 36?” The correct answer is 1,692. The model answered 1,656, which is wrong. Its perplexity (1.133) is only modestly elevated, close to the confident France answer. A wrong arithmetic answer does not necessarily look dramatically “unconfident” in its logprobs.
• The deliberately-fabricated “Chief Astronaut” prompt (where the prompt itself asks for a made-up but confident-sounding answer) produces a perplexity of 1.655, the highest of the four, which makes sense: the model is being asked to invent something with no grounding at all, and even though it complies fluently, the individual token choices (a name, a title) are inherently less predictable than a well-known fact.

The printer hallucination and the correct France answer are meaningfully different (1.473 vs 1.004). That looks promising. Step 6 shows why you cannot stop the analysis there.

Step 6: Combine the Three Layers Into One Pipeline

The First Version, and the Bug It Had

The obvious design is: run Layer 1, then Layer 2, then generate a grounded answer using the retrieved policy text, then reject the answer if Layer 3’s perplexity crosses some fixed threshold. The first version of this tutorial’s pipeline did exactly that, with a threshold of 1.35 (comfortably above the 1.004 baseline from Step 5). Testing it against real, correctly-grounded questions produced results like this:

Q: How many PTO days do I get after 3 years at the company?
Decision: ESCALATE_LOW_GENERATION_CONFIDENCE
Message: I found a possibly relevant policy but I'm not confident in
my answer: "After three years, you will accrue 20 paid time off (PTO)
days per year.". Please confirm with HR/IT.

That answer is completely correct. It matches the policy document word for word in meaning. But its measured perplexity was 1.693, above the naive threshold, so the pipeline wrongly escalated it as low-confidence. Digging into why revealed the real problem: the printer hallucination from Step 5 measured perplexity 1.473, lower than this correct PTO answer’s 1.693. A single global perplexity threshold cannot separate a fabrication from a correct answer here, because the two answers aren’t equally “predictable” for reasons that have nothing to do with correctness. The PTO answer contains a specific number (20) and a somewhat free choice of phrasing (“After three years” versus “Three years from now” versus a dozen other valid openings), both of which are inherently less predictable token-by-token than a well-worn fact like “Paris.” A correct, specific, well-grounded answer can legitimately have higher perplexity than a fluent, confident-sounding fabrication.

The Fix: Treat Layer 3 as a Signal, Not a Verdict

The fix is not a cleverer threshold, it’s a change in what Layer 3 is trusted to do. In the final pipeline below, Layer 3 always runs and its perplexity and minimum token logprob are always attached to the response, so a human reviewing logs can see them. But it only automatically escalates on a very high, deliberately conservative “sanity ceiling” (2.5, well above every correct, grounded answer measured while building this tutorial, which ranged from about 1.15 to 1.69), rather than trying to precisely separate correct from incorrect answers on its own. Layers 1 and 2, which run before generation and never depend on the model’s own self-reported confidence, are where the real filtering happens.

"""Combine all three layers into one confidence-aware answer pipeline.

Design note (learned empirically while building this file, see the tutorial's
Common Mistakes section): an early version auto-escalated any answer whose
perplexity crossed a single fixed threshold. That over-triggered on correct,
well-grounded answers (a correct PTO answer measured perplexity 1.693, higher
than a *fabricated* printer-model answer at 1.473) and under-triggered on the
fabrication. Layer 3 is kept here as an always-attached observability signal
and a conservative last-resort outlier gate, not the primary defense. Layers
1 and 2 do the real work because they run BEFORE generation.
"""
from layer1_boundary import boundary_check
from layer2_retrieval import retrieve
from layer3_logprobs import score_answer

# Set well above every "correct, grounded" perplexity observed during testing
# (1.15-1.69). This is a last-resort outlier catch, not a precision gate.
# In production, calibrate this against your own labeled eval set instead of
# reusing this number as-is.
PERPLEXITY_SANITY_CEILING = 2.5

GROUNDED_PROMPT_TEMPLATE = """Answer the question using ONLY the policy text below. Be concise.

Policy: {context}

Question: {question}
Answer:"""


def answer_question(question: str) -> dict:
    trace = {"question": question}

    boundary = boundary_check(question)
    trace["layer1_boundary"] = boundary
    if not boundary["in_domain"]:
        trace["decision"] = "OUT_OF_SCOPE"
        trace["message"] = "This is outside what I can help with. Try our general helpdesk instead."
        return trace

    retrieval = retrieve(question)
    trace["layer2_retrieval"] = retrieval
    if not retrieval["retrieval_confident"]:
        trace["decision"] = "ESCALATE_LOW_RETRIEVAL"
        trace["message"] = "I couldn't find a policy document I'm confident matches this question. Routing to a human."
        return trace

    grounded_prompt = GROUNDED_PROMPT_TEMPLATE.format(
        context=retrieval["matched_text"], question=question
    )
    generation = score_answer(grounded_prompt)
    trace["layer3_generation"] = generation

    if generation["perplexity"] > PERPLEXITY_SANITY_CEILING:
        trace["decision"] = "ESCALATE_OUTLIER_PERPLEXITY"
        trace["message"] = (
            f"My answer looked unusually uncertain (perplexity "
            f"{generation['perplexity']}), so I'm routing this to a human instead "
            f"of guessing."
        )
    else:
        trace["decision"] = "ANSWER"
        trace["message"] = generation["answer"]

    return trace


if __name__ == "__main__":
    test_questions = [
        "How do I connect to the VPN from home?",
        "How many PTO days do I get after 3 years at the company?",
        "What's the weather forecast for tomorrow?",
        "What is our policy on remote pair programming tools?",
        "Can I expense a $600 conference registration fee without approval?",
    ]
    for q in test_questions:
        result = answer_question(q)
        print("=" * 70)
        print("Q:", q)
        print("Decision:", result["decision"])
        if "layer3_generation" in result:
            g = result["layer3_generation"]
            print(f"  (perplexity={g['perplexity']}, min_token_logprob={g['min_token_logprob']})")
        print("Message:", result["message"])

Step 7: Run the Full Pipeline End to End

Running the pipeline against six real questions, including one deliberately tricky case at the end, produces this real, captured output:

======================================================================
Q: How do I connect to the VPN from home?
Decision: ANSWER
  Layer 1 boundary score: 0.7865 (closest: "How do I connect to the company VPN?")
  Layer 2 retrieval: doc=vpn-setup score=0.5958 margin=0.1429
  Layer 3 generation: perplexity=1.281 min_token_logprob=-2.309
Message: To connect to the VPN from home, follow these steps:

1. Access the internal software portal.
2. Install the GlobalProtect VPN client.
3. Log in using your company email and password.
4. Enable multi-factor authentication on first connection.
5. Contact IT if the installation fails.
======================================================================
Q: How many PTO days do I get after 3 years at the company?
Decision: ANSWER
  Layer 1 boundary score: 0.6784 (closest: "How many paid time off days do I get per year?")
  Layer 2 retrieval: doc=pto-policy score=0.7642 margin=0.2256
  Layer 3 generation: perplexity=1.693 min_token_logprob=-3.639
Message: After three years, you will accrue 20 paid time off (PTO) days per year.
======================================================================
Q: What's the weather forecast for tomorrow?
Decision: OUT_OF_SCOPE
  Layer 1 boundary score: 0.4207 (closest: "How do I request a new laptop or monitor?")
Message: This is outside what I can help with. Try our general helpdesk instead.
======================================================================
Q: What is our policy on remote pair programming tools?
Decision: OUT_OF_SCOPE
  Layer 1 boundary score: 0.4471 (closest: "How do I connect to the company VPN?")
Message: This is outside what I can help with. Try our general helpdesk instead.
======================================================================
Q: Can I expense a $600 conference registration fee without approval?
Decision: ANSWER
  Layer 1 boundary score: 0.5976 (closest: "Who approves expense reports over $500?")
  Layer 2 retrieval: doc=expense-limits score=0.5718 margin=0.1065
  Layer 3 generation: perplexity=1.243 min_token_logprob=-2.6277
Message: No, you cannot expense the $600 conference registration fee without approval. According to the policy, expenses over $500 require additional approval from a department director before reimbursement can be processed.
======================================================================
Q: What is the maximum number of sick days I can carry over between years?
Decision: ANSWER
  Layer 1 boundary score: 0.6371 (closest: "How many paid time off days do I get per year?")
  Layer 2 retrieval: doc=pto-policy score=0.605 margin=0.1216
  Layer 3 generation: perplexity=1.473 min_token_logprob=-2.0575
Message: The policy does not specify any maximum number of sick days that can be carried over between years. Therefore, based solely on the information provided, there is no clear answer regarding the maximum number of sick days that can be carried over.

Walk through what each one demonstrates:

• The VPN and PTO questions are correctly answered, with Layer 3’s perplexity attached for visibility but not blocking anything.
• The weather question is rejected at Layer 1 (boundary score 0.4207, below the 0.55 threshold), before any retrieval or generation ever happens.
• The pair-programming-tools question is also rejected at Layer 1 this time (boundary score 0.4471), rather than reaching Layer 2’s margin check as it did in the standalone Step 4 test. That’s expected and worth understanding: these layers are independent, overlapping defenses. In Step 4, Layer 2 was tested in isolation and caught the ambiguous match by its margin. In the full pipeline, Layer 1 happened to catch the same question first, on a different, cruder signal (this question doesn’t closely resemble any of the seven reference topics as a whole). Either layer catching a bad question is a success; you don’t need every layer to be the one that fires.
• The expense question is answered correctly and completely, citing the $500 approval threshold from the actual policy text.
• The sick-days question is the most important result. Layer 1 passes it (PTO-adjacent), Layer 2 confidently retrieves the PTO policy document (score 0.605, margin 0.1216, both well above threshold) because it genuinely is the closest topical match, and then the model, prompted to answer “using ONLY the policy text,” correctly notices the retrieved policy never mentions sick-day carryover at all and says so instead of inventing a number. This is the behavior you actually want: strong topical retrieval does not force a fabricated answer if the prompt makes refusal an explicit, easy option.

Step 8: Write Automated Tests

A pipeline like this deserves tests that would catch a regression in any layer. Here are seven, covering the boundary check, the margin-based retrieval catch, and two full end-to-end pipeline paths:

from layer1_boundary import boundary_check
from layer2_retrieval import retrieve
from pipeline import answer_question


def test_boundary_check_flags_out_of_domain():
    result = boundary_check("What's the weather like in Tokyo right now?")
    assert result["in_domain"] is False


def test_boundary_check_passes_in_domain():
    result = boundary_check("How do I set up the company VPN on my laptop?")
    assert result["in_domain"] is True


def test_retrieval_margin_catches_ambiguous_match():
    result = retrieve("What is our policy on remote pair programming tools?")
    assert result["passes_absolute_threshold"] is True
    assert result["passes_margin_threshold"] is False
    assert result["retrieval_confident"] is False


def test_retrieval_confident_on_clear_match():
    result = retrieve("How many vacation days do I earn after 3 years?")
    assert result["retrieval_confident"] is True
    assert result["matched_doc_id"] == "pto-policy"


def test_pipeline_rejects_out_of_scope_question():
    result = answer_question("What's the weather forecast for tomorrow?")
    assert result["decision"] == "OUT_OF_SCOPE"


def test_pipeline_answers_well_grounded_question():
    result = answer_question("How do I connect to the VPN from home?")
    assert result["decision"] == "ANSWER"
    assert "GlobalProtect" in result["message"]


def test_pipeline_refuses_to_fabricate_unsupported_detail():
    result = answer_question(
        "What is the maximum number of sick days I can carry over between years?"
    )
    assert result["decision"] == "ANSWER"
    assert "does not specify" in result["message"].lower() or "no clear answer" in result["message"].lower()

Running pytest test_pipeline.py -v against a live local Ollama server (these are not mocked; every test makes real embedding and generation calls) produced:

============================= test session starts =============================
platform win32 -- Python 3.13.14, pytest-9.1.1, pluggy-1.6.0
collecting ... collected 7 items

test_pipeline.py::test_boundary_check_flags_out_of_domain PASSED         [ 14%]
test_pipeline.py::test_boundary_check_passes_in_domain PASSED            [ 28%]
test_pipeline.py::test_retrieval_margin_catches_ambiguous_match PASSED   [ 42%]
test_pipeline.py::test_retrieval_confident_on_clear_match PASSED         [ 57%]
test_pipeline.py::test_pipeline_rejects_out_of_scope_question PASSED     [ 71%]
test_pipeline.py::test_pipeline_answers_well_grounded_question PASSED    [ 85%]
test_pipeline.py::test_pipeline_refuses_to_fabricate_unsupported_detail PASSED [100%]

============================= 7 passed in 52.31s ==============================

How to Verify Everything Works

1. Run ollama list and confirm both qwen2.5:1.5b and nomic-embed-text are present before running anything else.
2. Run each layer’s file on its own first (python layer1_boundary.py, then python layer2_retrieval.py, then python layer3_logprobs.py) and confirm the printed scores roughly match the ranges shown in this tutorial. Exact perplexity and logprob values can shift slightly between runs and Ollama versions (see the next section), but the ordering, in-domain scoring clearly higher than out-of-domain, correct grounded answers scoring reasonably, should hold.
3. Run python pipeline.py or python final_demo.py and confirm you get one OUT_OF_SCOPE, at least one ANSWER that is factually correct against your own policy text, and that a question your knowledge base cannot fully answer produces an honest “the policy does not specify” response rather than an invented one.
4. Run pytest test_pipeline.py -v and confirm all seven tests pass.
5. Try your own out-of-domain question (anything unrelated to your knowledge base) and confirm Layer 1 catches it before any generation happens; you can check this by adding a print statement or breakpoint right after the boundary_check call.

Common Mistakes and Gotchas

• A single similarity threshold is not enough for retrieval. As shown in Step 4, a genuinely irrelevant document can still clear an absolute similarity cutoff. Always also check the margin between the best and second-best match.
• Perplexity does not reliably separate hallucinations from correct answers on its own. This tutorial’s own data showed a correct PTO answer (perplexity 1.693) scoring less confident than a fabricated printer model number (perplexity 1.473). Treat token-level confidence as an observability signal and a conservative outlier catch, not a precision classifier, unless you calibrate it against a labeled dataset of correct and incorrect answers for your specific task.
• Grounded retrieval does not automatically prevent fabrication. The model still needs to be explicitly instructed to answer “using ONLY” the retrieved text. Without that instruction in the prompt, a model can pad a grounded answer with outside knowledge that isn’t actually verified.
• Ollama’s logprobs support is real but undocumented as of this writing. Ollama’s own compatibility docs list logprobs as unsupported on the OpenAI-compatible chat endpoint, while a pending documentation pull request and this tutorial’s own tests confirm it actually works. Check your specific Ollama version’s behavior rather than assuming either the docs or this tutorial are permanently accurate.
• Exact logprob and perplexity values are not perfectly reproducible run to run, even at temperature 0. While building this tutorial, the same grounded prompt for the PTO question, called in isolation five times in a row, returned identical output and identical perplexity every time. But the same prompt, called as part of a longer pipeline session after several other different Ollama requests, occasionally returned slightly different phrasing and a different perplexity value. This is consistent with (though not confirmed to be identical to) the floating-point non-determinism discussed in Ollama’s own issue tracker. Treat exact confidence numbers as approximate, not as a precise audit trail, unless you’ve specifically tested reproducibility on your own setup.
• Embeddings measure meaning, not correctness. Layer 1 and Layer 2 both rely entirely on semantic similarity. Neither one can tell you whether retrieved information is up to date or accurate, only whether it’s topically relevant. Keeping your knowledge base current is a separate problem this pipeline does not solve.

Next Steps

This pipeline uses fixed thresholds chosen by inspecting a handful of manually-run examples. A real production system should calibrate those thresholds against a labeled dataset of known-good and known-bad questions and answers for its own domain. sxz.io’s tutorial on evaluating RAG retrieval quality with precision, recall, and MRR covers exactly that methodology and pairs directly with the retrieval layer built here.

If you want to build the retrieval-augmented generation side of this out further, with a larger document set, chunking, and a full question-answering loop, see the site’s existing tutorial on building a local RAG Q&A agent with LangChain v1 and Ollama. And if you want the model’s output to be validated against a strict schema in addition to the confidence checks in this tutorial, see how to get reliable structured output from an LLM with validation and retries, which is a complementary technique: schema validation catches malformed output, while the layers in this tutorial catch fluent but untrustworthy output.

Other directions worth exploring on your own: log Layer 3’s perplexity and minimum token logprob over time in production and watch for drift, which can signal that your model or your traffic has changed in ways worth investigating. Consider replacing the raw perplexity check with a proper entailment or groundedness model that specifically checks whether an answer’s claims are supported by the retrieved text, rather than only measuring how “predictable” the answer’s wording was. And if you productionize this, pin Ollama to a specific version and re-run this tutorial’s own test suite against it before upgrading, given the documentation gap around logprobs described in Step 5.