How to Get Reliable Structured Output From an LLM With Validation and Retries

If you have ever asked a language model to “respond with valid JSON only” and then called json.loads() on the result, you already know how this story goes. It works for the first few requests. Then, sooner or later, the model wraps its answer in a markdown code fence anyway, invents a value for a field it should have left blank, or quietly assigns the wrong severity to a bug report. None of that shows up as an error you can catch with a try/except, because as far as the parser is concerned, nothing went wrong.

This tutorial builds a small, complete pipeline that extracts structured data from unstructured text using a local large language model (LLM), and teaches you the three layers you need to make that pipeline trustworthy: constraining the model’s output so it is valid JSON in the first place, validating that JSON against a schema so you catch the wrong shape, and a lightweight semantic check for the failures that are perfectly valid JSON but still wrong. Along the way you will build a retry loop that knows when to give up, and see, with real captured output from a real model, exactly which of these problems a retry can fix and which it cannot.

Every command in this tutorial runs against a free, local model through Ollama. There is no API key, no cloud bill, and no dependency on any specific hosted provider. The techniques (schema-first design, constrained decoding, two-layer validation, capped retries, grounding checks) apply just as well if you later swap in a hosted API.

What You Will Build

You will build a bug report extractor: a small program that takes a messy, freeform bug report (the kind a support inbox or a Slack channel is full of) and turns it into a structured record with a title, a severity level, a list of reproduction steps, and expected versus actual behavior. That specific task is not the point. The point is the reliability layer around it: schema validation, semantic validation, a retry loop, and a grounding check, all of which apply to any LLM extraction task you will ever build.

Prerequisites

• Ollama installed on Windows, macOS, or Linux. If you do not have it yet, install it from ollama.com/download, then pull two small models: ollama pull qwen2.5:1.5b and ollama pull qwen2.5:0.5b. This tutorial deliberately uses models in the 0.5 to 1.5 billion parameter range (small enough to run comfortably on a laptop CPU) because smaller models make more of the mistakes this tutorial is about catching, which makes them better teaching material than a larger model that rarely fails.
• Python 3.10 or later and pip.
• Basic comfort with the command line and with reading JSON. You do not need prior experience with Pydantic; this tutorial explains every piece of it as it comes up.
• No API keys and no cloud costs. Everything in this tutorial runs entirely on your own machine.

Create a project folder and an isolated virtual environment, then install the two libraries you need:

mkdir llm-extraction-tutorial
cd llm-extraction-tutorial
python -m venv venv
venv\Scripts\activate        # on Windows
# source venv/bin/activate   # on macOS/Linux

pip install pydantic requests

Confirm Ollama is running and has picked up your model:

curl http://localhost:11434/api/tags

You should see a JSON list of your locally pulled models. If the connection is refused, start the Ollama app (or run ollama serve) before continuing.

Step 1: Design the Schema First, Not the Prompt

The instinct when you need structured data from an LLM is to write a longer, more careful prompt. That is backwards. The more reliable habit is to write the schema first, in code, before you write a single word of prompt. A schema forces you to answer the ambiguous questions up front (is severity a free-text string or one of four fixed values? Is affected_component required or optional?) instead of discovering the ambiguity later when the model’s answer does not match what your application expected.

Create schema.py:

from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field, field_validator


class Severity(str, Enum):
    low = "low"
    medium = "medium"
    high = "high"
    critical = "critical"


class BugReport(BaseModel):
    title: str = Field(..., min_length=5, max_length=120)
    severity: Severity
    steps_to_reproduce: list[str] = Field(..., min_length=1, max_length=10)
    expected_behavior: str = Field(..., min_length=5)
    actual_behavior: str = Field(..., min_length=5)
    affected_component: Optional[str] = None

    @field_validator("steps_to_reproduce")
    @classmethod
    def steps_must_be_substantive(cls, v: list[str]) -> list[str]:
        for step in v:
            if len(step.strip()) < 3:
                raise ValueError(f"step '{step}' is too short to be a real instruction")
        return v


BUG_REPORT_JSON_SCHEMA = BugReport.model_json_schema()

Pydantic is a Python data-validation library: you describe a shape as a class with typed fields, and it checks any data you hand it against that shape, raising a detailed error if something does not match. A few choices here matter:

• Severity is an Enum with exactly four lowercase values. This is stricter than a free-text string field, and that strictness is the point: it gives the model (and your validation layer) a closed, unambiguous set of options instead of an open-ended one where “urgent,” “P1,” and “sev-2” are all plausible answers.
• min_length and max_length on title and the list fields reject degenerate answers (an empty title, zero reproduction steps) without you writing a single if statement.
• The field_validator on steps_to_reproduce is a custom rule Pydantic’s built-in type checks cannot express: not just “is this a list of strings” but “is each string long enough to actually be an instruction.”

Run it and look at the JSON schema Pydantic generates for you automatically:

python schema.py

(Add print(json.dumps(BUG_REPORT_JSON_SCHEMA, indent=2)) at the bottom of the file first, or run it interactively.) You will see a standard JSON Schema document with properties, a required list, and an embedded enum definition for Severity. You did not write that schema by hand. Pydantic derived it from your class definition, and in Step 3 you are going to hand that exact schema to the model.

Step 2: The Naive Approach Fails More Often Than You Would Think

Before reaching for anything clever, it is worth seeing the failure mode you are trying to avoid, on your own machine, with your own eyes. Save a few sample bug reports in sample_reports.py (the exact text does not matter; what matters is that they are messy and realistic):

CLEAN_REPORT = """
Subject: Export button does nothing on the Invoices page

When I click the "Export CSV" button on the Invoices page, nothing happens.
No download starts, no error message appears, no loading spinner shows up.
I expected clicking the button to start downloading a CSV file of the
current invoice list.

Steps I took:
1. Log in as a standard (non-admin) user
2. Go to Billing > Invoices
3. Click the "Export CSV" button in the top right
4. Wait about 30 seconds

This is blocking our monthly reconciliation, we can't get the data out
any other way right now. Affected area: the invoices/export module.
"""

AMBIGUOUS_SEVERITY_REPORT = """
The dashboard chart colors look slightly off compared to last week. The
blue bars used to be a darker navy and now they're more of a sky blue.
Everything else on the dashboard still loads and works fine. Just noticed
it this morning when I opened the analytics dashboard.
"""

GARBLED_STEPS_REPORT = """
ok so basically i was doing the checkout thing and like added stuff to cart
then went to pay and typed my card info in but then it just sat there
spinning forever and never finished, had to refresh and lost my cart, this
happened twice, using the checkout page, pretty annoying honestly and i
think it might be related to the new payment provider they switched to
last week
"""

HALLUCINATION_BAIT_REPORT = """
Users are reporting that the mobile app crashes immediately after opening
the Settings screen. This started happening after our last release. No
crash logs are available yet because the crash reporting SDK itself seems
to be failing to initialize. We don't know which OS or app version is
affected, we just have a handful of one-line complaints from the support
inbox.
"""

TYPO_REPORT = """
The tooltip that appears when you hover over the "Settings" gear icon says
"Setttings" with three T's instead of two. Everything else about the icon
and the menu it opens works exactly as expected. Just a spelling mistake.
"""

ALL_REPORTS = {
    "clean": CLEAN_REPORT,
    "ambiguous_severity": AMBIGUOUS_SEVERITY_REPORT,
    "garbled_steps": GARBLED_STEPS_REPORT,
    "hallucination_bait": HALLUCINATION_BAIT_REPORT,
    "typo": TYPO_REPORT,
}

Now write a small Ollama client helper in ollama_client.py. Set temperature: 0 (always pick the single most likely next token instead of sampling) and a fixed seed from the start, so that every result in this tutorial, including the failures, is exactly reproducible on your own machine:

import json
import requests

OLLAMA_URL = "http://localhost:11434/api/chat"


def chat(model: str, prompt: str, format_schema: dict | None = None) -> dict:
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": {"temperature": 0, "seed": 42},
    }
    if format_schema is not None:
        payload["format"] = format_schema
    resp = requests.post(OLLAMA_URL, json=payload, timeout=120)
    resp.raise_for_status()
    return resp.json()

Every result quoted in this tutorial was rerun twice with this exact configuration to confirm the output was byte-for-byte identical both times before being included here. If you skip pinning temperature and seed, you will still see the same classes of failure demonstrated below, just with different specific wording each time, since Ollama samples somewhat randomly by default.

And the naive extraction attempt, asking politely for JSON with no structural constraint at all:

import json
from ollama_client import chat
from sample_reports import ALL_REPORTS

MODEL = "qwen2.5:1.5b"

PROMPT_TEMPLATE = """Extract the bug report below into JSON with exactly these keys:
title, severity (one of: low, medium, high, critical), steps_to_reproduce
(a list of strings), expected_behavior, actual_behavior, affected_component.

Respond with valid JSON only, no other text.

Bug report:
{text}
"""

for name, text in ALL_REPORTS.items():
    prompt = PROMPT_TEMPLATE.format(text=text)
    result = chat(MODEL, prompt)
    content = result["message"]["content"]
    print(f"=== {name} ===")
    print(content)
    try:
        json.loads(content)
        print("-> json.loads() OK")
    except json.JSONDecodeError as e:
        print(f"-> json.loads() FAILED: {e}")

Running this against a real local qwen2.5:1.5b model produced this, verbatim, for every single one of the five test reports:

=== clean ===
```json
{
  "title": "Export button does nothing on the Invoices page",
  "severity": "critical",
  "steps_to_reproduce": [
    "Log in as a standard (non-admin) user",
    "Go to Billing > Invoices",
    "Click the 'Export CSV' button in the top right"
  ],
  "expected_behavior": "The export should start downloading a CSV file of the current invoice list.",
  "actual_behavior": "No download starts, no error message appears, no loading spinner shows up.",
  "affected_component": "the invoices/export module"
}
```

-> json.loads() FAILED: Expecting value: line 1 column 1 (char 0)

All five reports, despite the explicit instruction “respond with valid JSON only, no other text,” came back wrapped in a Markdown code fence (three backticks and the word json). json.loads() chokes on the leading backticks and fails immediately, every time. The model is not being uncooperative. It has simply seen an enormous amount of training data where JSON appears inside a fenced code block, and that pattern is strong enough to override a one-line instruction most of the time. This is exactly the kind of failure you cannot fix by writing a firmer sentence into the prompt. You need the API to enforce the shape structurally instead of asking nicely.

Step 3: Constrain the Output With a JSON Schema

Ollama’s chat endpoint accepts a format parameter with two distinct modes, and the difference between them matters:

• format: "json" (JSON mode) only guarantees the response is some syntactically valid JSON object. It says nothing about which keys it has or what type they are.
• format: <a JSON schema> (structured outputs, sometimes called constrained decoding) masks the model’s token sampler at every step so that a token which would violate the schema cannot be produced at all. Invalid output is not just unlikely, it is structurally impossible.

You already have a JSON schema sitting in BUG_REPORT_JSON_SCHEMA from Step 1. Pass it straight through:

import json
from ollama_client import chat
from sample_reports import ALL_REPORTS
from schema import BUG_REPORT_JSON_SCHEMA, BugReport
from pydantic import ValidationError

MODEL = "qwen2.5:1.5b"
PROMPT_TEMPLATE = """Extract the bug report below into the required fields.

Bug report:
{text}
"""

for name, text in ALL_REPORTS.items():
    prompt = PROMPT_TEMPLATE.format(text=text)
    result = chat(MODEL, prompt, format_schema=BUG_REPORT_JSON_SCHEMA)
    content = result["message"]["content"]
    data = json.loads(content)          # this no longer fails
    report = BugReport.model_validate(data)  # neither does this
    print(f"{name}: severity={report.severity}")

Run it, and every one of the five reports now parses and validates cleanly on the first try. No code fence, no retries, no exceptions:

clean: severity=Severity.critical
ambiguous_severity: severity=Severity.medium
garbled_steps: severity=Severity.medium
hallucination_bait: severity=Severity.critical
typo: severity=Severity.low

Notice what changed and what did not: the prompt barely mentions the required fields anymore, because the schema itself now carries that information. The markdown-fence problem, which broke 5 out of 5 responses in Step 2, is completely gone. That is a real, structural win, and it costs nothing beyond passing the schema you already had. But look closely at the severity values above. Is a spelling mistake in a tooltip really “low,” matching what you would hope? And is the export-button bug (annoying, but presumably the user can still see their invoices some other way) really “critical”? Schema validation has nothing to say about that question, because every one of those five answers is, as far as the schema is concerned, an equally valid string from the allowed set. That is the subject of the next step.

Step 4: Validation Is Two Different Jobs, Not One

It is tempting to think of “validation” as a single pass: does the data match the schema, yes or no. In practice it is two separate jobs that are easy to accidentally conflate:

• Structural validation (what Pydantic just did for you): does the shape match? Right keys, right types, right constraints.
• Semantic validation: does the content make sense, given what the model was actually told? A severity of “critical” is structurally valid every single time. Whether it is the correct severity for a one-character typo is a different question entirely, and no amount of schema tightening will ever answer it, because the schema does not know what the source text said.

Semantic checks are necessarily heuristic (you are encoding a business rule, not a type system), but a simple one already catches real problems. Here is a keyword-based sanity check: a report should not be marked “high” or “critical” unless the text describing what actually happened contains some concrete evidence of impact.

from schema import BugReport, Severity

IMPACT_KEYWORDS = [
    "crash", "cannot", "can't", "won't", "unable", "fail", "block",
    "lost", "lose", "down", "outage", "security", "data loss",
    "stuck", "never", "forever", "doesn't work", "does not work",
]


class SemanticWarning(Exception):
    """Schema-valid, but fails a business-rule sanity check."""


def check_severity_has_impact_evidence(report: BugReport) -> None:
    if report.severity not in (Severity.high, Severity.critical):
        return
    haystack = (report.actual_behavior + " " + " ".join(report.steps_to_reproduce)).lower()
    if not any(kw in haystack for kw in IMPACT_KEYWORDS):
        raise SemanticWarning(
            f"severity={report.severity.value} was assigned, but actual_behavior "
            f"contains no impact keyword. actual_behavior was: {report.actual_behavior!r}"
        )

Run this check against real, schema-constrained extractions from qwen2.5:0.5b, and it fires on one of the five, the “clean” export-button report:

=== clean ===  severity=critical
-> SEMANTIC CHECK FAILED: severity=critical was assigned, but actual_behavior/
steps_to_reproduce contain no impact keyword ([...]). actual_behavior was:
"Nothing happens when clicking the 'Export CSV' button."

=== ambiguous_severity ===  severity=medium   -> semantic check OK
=== garbled_steps ===  severity=medium        -> semantic check OK
=== hallucination_bait ===  severity=critical -> semantic check OK
=== typo ===  severity=low                    -> semantic check OK

This is worth reading twice, because it teaches something more specific than “the model got the severity wrong.” critical is arguably a defensible severity for the clean report; the full source text does say this is “blocking our monthly reconciliation” and that the user “can’t get the data out any other way right now.” The problem is that the model’s own actual_behavior field, the exact text it chose to extract, dropped that justification entirely and kept only “nothing happens when clicking the button.” A support engineer who only ever sees the four structured fields, never the original ticket, would have no way to tell why this was marked critical instead of medium. The semantic check does not know whether critical is objectively correct; it only knows whether the model’s own extracted evidence supports the severity the model itself assigned, and here it does not. That gap, between the label and the evidence for the label, is precisely the kind of problem structural validation cannot see, because both the label and the evidence are individually well-formed strings.

Be honest with yourself about what this heuristic is and is not. It is a keyword match against the model’s own output, not a full understanding of the original text. It will have false positives (a genuinely critical bug whose extracted description happens not to use one of these particular words) and false negatives (a trivial bug whose extracted description happens to include a word like “fail” in an unrelated sentence). That is fine. A cheap, deterministic sanity check that flags the obvious mismatches for a human to glance at is still far better than nothing standing between the model’s severity field and your ticket queue.

Step 5: Build a Retry Loop That Knows When to Stop

Now combine what you have. When json.loads() or Pydantic’s validation raises an error, that error message is itself useful: it names the exact field and the exact problem. Feed it back to the model and ask it to fix that specific thing, rather than starting over from scratch or, worse, looping forever.

import json
import re
from ollama_client import chat
from schema import BugReport
from pydantic import ValidationError

FENCE_RE = re.compile(r"^```(?:json)?\s*|\s*```$")


def strip_fence(text: str) -> str:
    return FENCE_RE.sub("", text.strip())


BASE_PROMPT = """Extract the bug report below into JSON with exactly these keys:
title, severity, steps_to_reproduce (a list of strings), expected_behavior,
actual_behavior, affected_component.

For severity, use a single word describing how bad the issue is.

Respond with JSON only.

Bug report:
{text}
"""


def extract_with_retry(model: str, source_text: str, max_attempts: int = 3):
    history = []
    correction = ""
    for attempt in range(1, max_attempts + 1):
        prompt = BASE_PROMPT.format(text=source_text) + correction
        result = chat(model, prompt)
        raw = strip_fence(result["message"]["content"])

        try:
            data = json.loads(raw)
        except json.JSONDecodeError as e:
            history.append((attempt, "json_error", str(e)))
            correction = f"\n\nYour last response was not valid JSON ({e}). Reply with JSON only, no markdown fences."
            continue

        try:
            report = BugReport.model_validate(data)
            history.append((attempt, "ok", None))
            return report, attempt, history
        except ValidationError as e:
            history.append((attempt, "validation_error", str(e)))
            correction = f"\n\nYour last response failed validation:\n{e}\nFix the field(s) named above and reply with corrected JSON only."
            continue

    raise RuntimeError(f"gave up after {max_attempts} attempts: {history}")

This version deliberately uses a loose, unconstrained prompt (no format_schema) rather than Step 3’s schema-constrained version, because that is what makes the retry loop’s job non-trivial and worth watching. Running it against the smallest model, qwen2.5:0.5b, on all five reports:

=== clean ===
-> succeeded on attempt 2/3, severity=high
   attempt 1: validation_error: 1 validation error for BugReport

=== ambiguous_severity ===
-> succeeded on attempt 2/3, severity=low
   attempt 1: validation_error: 1 validation error for BugReport

=== garbled_steps ===
-> FAILED: gave up after 3 attempts: [
  (1, 'validation_error', "severity: Input should be 'low', 'medium', 'high'
      or 'critical' [input_value='High', input_type=str]"),
  (2, 'validation_error', "severity: Input should be 'low', 'medium', 'high'
      or 'critical' [input_value='High', input_type=str]"),
  (3, 'validation_error', "severity: Input should be 'low', 'medium', 'high'
      or 'critical' [input_value='High', input_type=str]")
]

=== hallucination_bait ===
-> succeeded on attempt 3/3, severity=high

=== typo ===
-> succeeded on attempt 2/3, severity=high

Read the garbled_steps failure carefully, because it is the most instructive result here, and because the fixed temperature and seed from Step 2 make it reproducible rather than a fluke: all three attempts failed with the exact same error, input_value='High', capitalized, when the schema only accepts the lowercase high. The error message told the model precisely which field was wrong and precisely which four lowercase values were allowed, three times in a row, and the model sent back the identical capitalized string every single time. It did not converge, and at temperature zero it provably never would have, no matter how many more attempts you allowed. This is exactly why the loop is capped at a fixed number of attempts and raises a clear, inspectable error with the full history rather than retrying forever: a model that cannot act on a specific, correct piece of feedback after one attempt frequently cannot act on it after ten either, and every attempt costs real tokens. In production, the right response to catching this RuntimeError is to log the full attempt history and escalate to a human reviewer or a larger model, not to raise the retry ceiling and hope the next attempt is different.

Step 6: The Failures No Retry Can Fix

Retries only help when the model can produce a correct answer but formatted it wrong. They do nothing when the model’s understanding of the content itself is wrong, because there is no exception to catch, no field out of range, nothing for Pydantic to object to. What actually helps for this class of problem is a grounding check: require the model to also produce a short, verbatim quote from the source text supporting its claim, then verify programmatically that the quote actually exists in the source.

import re

GROUNDED_SCHEMA = {
    "type": "object",
    "properties": {
        "severity": {"type": "string", "enum": ["low", "medium", "high", "critical"]},
        "actual_behavior": {"type": "string"},
        "evidence_quote": {
            "type": "string",
            "description": "A short phrase copied verbatim from the bug report that justifies actual_behavior.",
        },
    },
    "required": ["severity", "actual_behavior", "evidence_quote"],
}


def normalize(s: str) -> str:
    return re.sub(r"\s+", " ", s.lower().strip())


def is_grounded(quote: str, source_text: str) -> bool:
    return normalize(quote) in normalize(source_text)

A Real Catch: Fabricated Text Presented as a Quote

Ask qwen2.5:0.5b for evidence_quote alongside its extraction on the garbled_steps report (a real, messy description of a checkout page that hangs after switching payment providers), and it produced this, reproducibly, at temperature zero:

=== garbled_steps === severity=critical
  evidence_quote: 'thishappenedtwice,usingthecomponentcheckoutpage,prettyannoyinghardlyandi
  thinkitmightberelevatedbythenewpaymentprovidertheyswitchedtolastweek'
  grounded in source text: False

Compare that quote to the actual source sentence: “this happened twice, using the checkout page, pretty annoying honestly and i think it might be related to the new payment provider they switched to last week.” The model’s “quote” inserted a word that is not there (“component”), swapped “honestly” for “hardly,” and mangled “related to” into “relevated by.” None of that is a hallucinated fact exactly, but it is fabricated text being presented as a verbatim citation, and the is_grounded check catches it immediately, because the string genuinely does not exist anywhere in the source, no matter how you normalize whitespace and case.

A False Alarm: When the Check Itself Is Too Strict

Now look at what the same check did with the typo report:

=== typo === severity=high
  evidence_quote: "The tooltip that appears when you hover over the 'Settings' gear icon
  says 'Setttings' with three T's instead of two. Everything else about the icon and
  the menu it opens works exactly as expected."
  grounded in source text: False

This one is instructive for a completely different reason: it is a false alarm. Compare the quote to the source text, which reads The tooltip... says "Settings" with three T's... using straight double quotes around the word, while the model’s quote used single quotes instead. The content is accurate, nothing is invented, but normalize() only lowercases text and collapses whitespace; it does not unify quote-mark styles, so a byte-for-byte substring check correctly (by its own narrow definition) reports “not grounded” for a quote that is, in substance, completely faithful to the source.

Put those two results side by side and you get the honest picture of this technique. It caught a real fabrication in garbled_steps: a true positive worth having. It also flagged a faithful quote in typo purely because of quote-character style: a false positive you would need to fix (normalizing quote characters, or switching to a fuzzy match tolerant of small formatting differences, rather than exact substring containment) before trusting this check unattended in production. The lesson is not “grounding checks are perfect.” It is that verifying a claim against the source text is a fundamentally different, complementary signal to schema and keyword validation, one that catches a class of problem neither of the other two can see, provided you are honest with yourself about its own rough edges.

Step 7: Structured Output and Streaming Do Not Mix the Way You Would Expect

Ollama’s chat endpoint supports streaming ("stream": true), which returns the response one small chunk at a time instead of waiting for the whole thing. Streaming feels responsive for plain text, but it creates a specific trap for structured output: your instinct might be to try parsing the growing buffer as JSON after every chunk, so you can react to the data as soon as it looks complete. In practice, that does not work, and it is worth measuring exactly how often it does not work rather than assuming.

import json
from ollama_client import chat_stream  # a streaming variant of chat(), same idea with stream=True

buffer = ""
early_successes = 0
non_final_attempts = 0

for chunk in chat_stream(MODEL, prompt, format_schema=BUG_REPORT_JSON_SCHEMA):
    buffer += chunk.get("message", {}).get("content", "")
    if not chunk.get("done"):
        non_final_attempts += 1
        try:
            json.loads(buffer)
            early_successes += 1
        except json.JSONDecodeError:
            pass

print(f"early successes: {early_successes} / {non_final_attempts}")

Across a real run against qwen2.5:1.5b, this streamed 118 chunks for a single extraction. Out of 117 non-final chunks, exactly 1 happened to parse as valid JSON early, and that one was already at 99 percent of the final buffer’s length, essentially the object closing its last brace a fraction of a second before the stream officially ended. The other 116 attempts failed, because a partial JSON object (an open string, a dangling comma, a bracket that has not been closed yet) is not valid JSON by definition, no matter how far along it is. The practical answer is not to fight this: buffer the full stream, wait for done: true, and parse once. If you genuinely need partial structured data before the response finishes (for a live-updating UI, for example), that requires a dedicated incremental JSON parser designed for exactly this problem, not repeated calls to the standard library’s all-or-nothing json.loads().

Common Mistakes and Gotchas

• Trusting “respond with JSON only” as your only safeguard. As Step 2 showed, this instruction competes with the model’s training and loses often enough to matter. Use format with a real JSON schema whenever the runtime supports it.
• Treating schema validation as the whole job. A schema-valid response can still be semantically wrong, overconfident, or backwards. Build at least one content-level check, even a simple keyword heuristic, for anything downstream that matters.
• Retrying without a cap. An uncapped retry loop against a model that fundamentally cannot solve the task will burn tokens forever. Cap attempts, log the full history on failure, and escalate instead of looping.
• Assuming a retry fixes what a validation error did not cause. Retries repair format problems the model can plausibly see and correct. They do nothing for a fact the model quietly invented, because there is no error message pointing at it.
• Parsing a streaming buffer incrementally with a standard JSON parser. It will silently fail almost every time until the object is complete. Buffer until done, then parse once.

Verify It All Works End to End

Put the pieces together and run the full pipeline against all five sample reports: schema-constrained extraction, Pydantic validation, the semantic keyword check, the retry loop with its cap, and the grounding check. With temperature: 0 and a fixed seed as set up back in Step 2, you should see:

• Every report either succeeds within 3 attempts or raises a clear RuntimeError naming the exact fields that kept failing, and running the whole pipeline a second time reproduces the identical result, attempt counts included.
• The semantic check flags at least the “clean” report, because its own extracted actual_behavior does not carry the impact language that would justify the severity assigned to it, even though the object is fully schema-valid.
• The grounding check flags at least the garbled_steps report’s evidence quote as not present in the source text, a real, reproducible sign that specific words were fabricated rather than copied.
• If you remove the options: {"temperature": 0, "seed": 42} block, rerunning the pipeline several times still produces the same classes of outcome (some reports pass cleanly, some get flagged, occasionally one exhausts its retries) but with different specific wording and different severities each time, since Ollama samples somewhat randomly by default. That variability is itself a reason to keep the validation and semantic layers in place even after you have a demo that looks like it works: a run that happens to look clean once is not the same as a pipeline that is reliably correct.

If you see all of that, your pipeline is doing its job: catching the syntax problems for free, catching a class of semantic problems on purpose, catching at least one class of fabricated content the first two layers cannot see, and failing loudly and cheaply on the rest instead of quietly shipping bad data.

Next Steps

Everything here used a local model through Ollama because it costs nothing and keeps every example fully reproducible. The same three-layer idea (constrain, validate twice, retry with a cap) carries over directly to hosted APIs, under different names: OpenAI’s API offers a structured outputs feature that constrains a response to match a JSON schema you supply, playing the same role as Ollama’s schema-constrained format. Anthropic’s Messages API offers a similar native structured-output mode via its output_config request field, plus a strict: true option on individual tool definitions that guarantees a tool call’s arguments validate exactly against that tool’s schema. In every case, the underlying lesson from this tutorial does not change: constraining the output format eliminates syntax failures, but it is a separate, deliberate choice to also validate meaning, and an even more deliberate choice to check that a claim is actually grounded in your source text instead of merely plausible-sounding.

From here, a natural next step is to plug this pipeline into a real data source (a support ticket queue, an email inbox, a log stream) and track, over a week of real traffic, how often the semantic check and the grounding check actually fire. That number will tell you far more about whether your extraction pipeline is trustworthy than any single demo run can.