How to Calibrate an LLM-as-Judge Against Human Ratings With Cohen’s Kappa

If you have ever built an automated grader for an AI system, an “LLM-as-judge” that reads a response and decides whether it is good enough, you have probably asked yourself a harder question afterward: how do you know the judge itself is any good? A judge that quietly disagrees with what a real person would say is worse than no judge at all, because it gives you false confidence. This tutorial teaches you how to answer that question properly: how to validate an LLM judge against real human ratings using a statistic called Cohen’s kappa, and how to tell whether a change you made to the judge’s prompt actually improved it, or whether the apparent improvement is just noise from too small a test set.

Everything in this post is built from scratch in plain Python and run against a local model through Ollama, so there is no API key to manage and no cost per run. By the end you will have a reusable “calibration report” you can drop into your own evaluation pipeline, and you will understand exactly what it is telling you and what it is not.

What You’ll Need

• Python 3.10 or newer, with pip.
• Ollama installed and running locally. If you have not set this up yet, How to Install Ollama and Run Your First Local AI Chatbot walks through it step by step.
• A small local model pulled. This tutorial uses qwen2.5:1.5b, which runs comfortably on CPU: ollama pull qwen2.5:1.5b.
• Basic comfort calling an HTTP API from Python. If you want the background on building an LLM judge in the first place, see How to Build a Local LLM-as-a-Judge Evaluation Harness for AI Agents, though this post does not assume you have read it.

Install the two Python packages we need beyond the standard library:

pip install requests pytest

Why “It Agreed With Me Once” Is Not Validation

Say you build an LLM judge, try it on five examples, and it gets all five “right” by your own read of them. That feels like validation, but it is really just one person’s opinion checked against itself. Two things are missing.

First, you need a second, independent opinion. If you are the only person who ever looks at the judge’s output, you cannot tell the difference between “the judge is well calibrated” and “the judge happens to share my particular blind spots.” Real teams use more than one reviewer for exactly this reason: it exposes disagreement that a single reviewer would never notice in themselves.

Second, once you have two reviewers, you need a way to measure how much they agree that accounts for luck. If two people are rating mostly-good outputs, they will agree a lot of the time purely by chance, even if their actual judgment differs. The statistic that corrects for this is called Cohen’s kappa, and it is the core tool this tutorial builds from scratch.

Step 1: Build a Small Golden Set of Real AI Responses

To make this concrete, we will build a tiny fictional support bot for a fictional product called CloudNest, a cloud file-sync and note-taking app. The bot answers customer questions using a fixed set of reference facts, similar to a small internal knowledge base.

kb.py holds the facts and the question set:

KB_FACTS = """
CloudNest is a cloud file-sync and note-taking product. Support facts:
- Free plan: 5GB storage, 1 device.
- Pro plan: $9/month, 200GB storage, unlimited devices.
- Team plan: $19/user/month, 1TB pooled storage per user, includes API access.
- Maximum file size for any single upload: 5GB.
- Supported apps: Windows, macOS, Linux, iOS, and Android.
- Two-factor authentication (2FA) is supported using TOTP authenticator apps.
- Shared links can have an optional password and an optional expiry date.
- Offline sync is supported on all desktop and mobile apps.
- Deleted files are recoverable from the trash for 30 days, then permanently erased.
- Team plan support response SLA is 24 hours.
"""

SYSTEM_PROMPT = (
    "You are the CloudNest customer support assistant. Answer the customer's "
    "question using ONLY the facts below. Be concise (2-3 sentences). If the "
    "answer is not covered by these facts, say you don't have that information "
    "rather than guessing.\n" + KB_FACTS
)

Ten of the sixteen questions are answerable directly from those facts. The other six are deliberately not covered by the knowledge base, things like “do you offer a discount for annual billing” or “what encryption algorithm do you use.” A good answer to those admits the gap. A bad answer invents a confident-sounding specific that is not actually backed by anything.

Now generate real answers with a local model, at temperature 0 so the run is reproducible:

OLLAMA_URL = "http://127.0.0.1:11434/api/chat"
MODEL = "qwen2.5:1.5b"

def ask(question: str) -> str:
    payload = {
        "model": MODEL,
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
        "stream": False,
        "options": {"temperature": 0, "seed": 42},
    }
    r = requests.post(OLLAMA_URL, json=payload, timeout=120)
    r.raise_for_status()
    return r.json()["message"]["content"].strip()

Running this against all sixteen questions produced these real, unedited responses. The first ten (in the knowledge base) all came back correct:

[ 0] How much storage does the free plan include?
     The free plan includes 5GB of storage.

[ 8] What operating systems do you support?
     CloudNest supports Windows, macOS, Linux, iOS, and Android.

[ 9] How much does the Pro plan cost per month?
     The Pro plan costs $9/month.

The last six, outside the knowledge base, are where things get interesting. This is real model output, not staged:

[10] Do you offer a discount for annual billing?
     No, CloudNest does not offer an annual billing discount.

[11] What encryption algorithm do you use for files at rest?
     I don't have specific information about CloudNest's file encryption
     algorithms. However, it supports the AES (Advanced Encryption Standard)
     algorithm which provides strong security for data protection.

[12] Can I migrate my files automatically from Dropbox?
     No, CloudNest does not offer automatic migration of files from Dropbox.
     You would need to manually transfer your data using the CloudNest app
     or website.

[13] Do you support SAML single sign-on for enterprise customers?
     No, CloudNest does not support SAML single sign-on for enterprise
     customers.

[14] What's your uptime SLA guarantee, as a percentage?
     The team plan support response SLA is 24 hours.

[15] Is there a command-line interface for Linux, or only a desktop app?
     CloudNest supports both Windows and macOS. It does not have a built-in
     command-line interface specifically designed for Linux. However, you
     can use the CloudNest desktop application on your Linux system to
     access and manage your files.

Look closely at item 11: the model correctly hedges (“I don’t have specific information”) and then immediately contradicts itself by asserting AES anyway. Item 14 answers a completely different question than the one asked. Item 15 claims CloudNest supports “both Windows and macOS” while ignoring the Linux, iOS, and Android support already stated two questions earlier, then talks about “your Linux system” in the very next sentence. None of this was engineered; it is what a small model does when you ask it things outside its grounding.

Step 2: Have Two Humans Rate the Responses (and Let Them Disagree)

To validate a judge you need real human labels to check it against. Rather than fabricate a single “correct” answer for each item, this tutorial uses two distinct, disclosed rating rubrics applied by hand to the sixteen real responses above. This is not a live crowdsourced study; it is a worked example of how two reasonable reviewers, using different but legitimate standards, land on different verdicts.

• Rater A, “Strict-Sam”: fails any response that states a specific fact not present in the reference facts, even if it sounds plausible, or that is off-topic or self-contradictory. Passes only responses that stick to the facts or honestly admit a gap.
• Rater B, “Lenient-Lee”: passes any response a real customer would likely find helpful and non-harmful, even if it includes a plausible unverified detail. Fails only responses that are actively unhelpful, off-topic, or confusing enough to hurt the customer.

Applying both rubrics to the same sixteen items by hand:

[ 0] A=PASS B=PASS      [ 8] A=PASS B=PASS
[ 1] A=PASS B=PASS      [ 9] A=PASS B=PASS
[ 2] A=PASS B=PASS      [10] A=FAIL B=PASS  <-- disagreement
[ 3] A=PASS B=PASS      [11] A=FAIL B=PASS  <-- disagreement
[ 4] A=PASS B=PASS      [12] A=FAIL B=PASS  <-- disagreement
[ 5] A=PASS B=PASS      [13] A=FAIL B=FAIL
[ 6] A=PASS B=PASS      [14] A=FAIL B=FAIL
[ 7] A=PASS B=PASS      [15] A=FAIL B=FAIL

All ten in-knowledge-base answers get a clean pass from both raters. Of the six out-of-knowledge-base answers, the two raters agree on three (13, 14, and 15, all clearly bad) and disagree on three (10, 11, and 12, where Strict-Sam flags the invented specifics and Lenient-Lee is willing to forgive them as generically helpful).

Step 3: Percent Agreement Is a Trap

The obvious first check is: how often do the two raters agree? That is percent agreement, and it is exactly what it sounds like:

def percent_agreement(labels_a, labels_b):
    matches = sum(1 for a, b in zip(labels_a, labels_b) if a == b)
    return matches / len(labels_a)

Running it on Rater A and Rater B’s sixteen labels:

>>> percent_agreement(rater_a, rater_b)
0.8125

81.25 percent agreement sounds solid. It is also misleading, and the next section shows exactly why.

Step 4: Cohen’s Kappa, Built From Scratch

The problem with percent agreement is that it does not account for agreement you would expect by pure chance. If one rater passes almost everything, the other rater will “agree” with a lot of those passes just by also tending to pass things, independent of whether their judgment actually lines up.

The Confusion Matrix

Start by counting how the two raters’ verdicts line up:

              Rater B: PASS   Rater B: FAIL
Rater A: PASS       10              0
Rater A: FAIL         3              3

Ten items where both said PASS, three where A said FAIL but B said PASS, and three where both said FAIL. That is 13 out of 16 matching cells, the same 0.8125 as before.

Expected Agreement by Chance

Cohen’s kappa first works out how often these two raters would agree if their labels were assigned independently of each other, based only on how often each rater uses each label overall. Rater A said PASS on 10 of 16 items (62.5 percent) and FAIL on 6 (37.5 percent). Rater B said PASS on 13 of 16 (81.25 percent) and FAIL on 3 (18.75 percent). If their choices were statistically independent, the expected chance of both saying PASS is 0.625 times 0.8125, and the expected chance of both saying FAIL is 0.375 times 0.1875:

def cohens_kappa(labels_a, labels_b):
    n = len(labels_a)
    categories = sorted(set(labels_a) | set(labels_b))
    po = sum(1 for a, b in zip(labels_a, labels_b) if a == b) / n
    count_a = Counter(labels_a)
    count_b = Counter(labels_b)
    pe = sum((count_a[c] / n) * (count_b[c] / n) for c in categories)
    if pe == 1.0:
        return 1.0
    return (po - pe) / (1 - pe)

The Kappa Formula

The formula itself, as defined on Wikipedia’s Cohen’s kappa reference, is kappa equals (p_o minus p_e) divided by (1 minus p_e), where p_o is the observed agreement (0.8125, computed above) and p_e is the expected chance agreement:

p_e = 0.625 * 0.8125 + 0.375 * 0.1875 = 0.578125
kappa = (0.8125 - 0.578125) / (1 - 0.578125) = 0.5556

Running the actual function confirms the hand calculation:

>>> cohens_kappa(rater_a, rater_b)
0.5556

The raw agreement said 81 percent. Once you correct for how often these two raters would have agreed by chance alone, given how skewed their individual PASS rates already are, the real agreement is a noticeably weaker 0.5556. That gap between 0.8125 and 0.5556 is entirely explained by Rater B’s heavy lean toward PASS, not by any genuine additional agreement between the two.

Reading the Kappa Scale

Kappa values are commonly interpreted using a scale from Landis and Koch’s 1977 paper, summarized on the same Wikipedia reference: below 0 is no agreement, 0 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1.0 is almost perfect. Worth knowing before you lean on it: this scale is a widely used convention, not a mathematically derived standard, and the original authors offered it as their own judgment rather than backing it with independent evidence. Treat it as a rough vocabulary, not a pass or fail line.

By that scale, our two human raters land at 0.5556, “moderate,” despite 81 percent of their raw verdicts matching. That moderate number, not the 81 percent, is the honest description of how much these two reviewers actually agree.

Step 5: The Accuracy Paradox, Isolated

To see how bad this gap can get, try a deliberately lazy judge on a separate, clearly synthetic example: a judge that always says PASS, no matter what, checked against a ground truth where 18 of 20 items are genuinely PASS and 2 are genuinely FAIL:

ground_truth = ["PASS"] * 18 + ["FAIL"] * 2
lazy_judge = ["PASS"] * 20

>>> percent_agreement(lazy_judge, ground_truth)
0.9000
>>> cohens_kappa(lazy_judge, ground_truth)
0.0000

A judge that does zero work, that never even looks at the input, scores 90 percent on percent agreement simply because most items happen to be PASS. Kappa correctly reports exactly zero: this judge has no discriminating power whatsoever. Any time your evaluation results are reported only as a raw agreement or accuracy percentage against a skewed label set, be suspicious. This is exactly the failure mode kappa exists to catch.

Step 6: Build the LLM-as-Judge, Two Versions

Now build an actual automated judge and see how it stacks up against the two human raters. We will test two prompt versions on the same local model.

Judge v1 is deliberately minimal, no rubric, no reference facts, just a bare instruction:

JUDGE_V1_SYSTEM = (
    "You are grading a customer support response. Answer with exactly one "
    "word: PASS or FAIL."
)

Judge v2 gets the full reference facts plus an explicit rubric that mirrors Strict-Sam’s own criteria:

JUDGE_V2_SYSTEM = (
    "You are grading a customer support response against a fixed set of "
    "reference facts. Here are the only facts you may treat as confirmed:\n"
    + KB_FACTS +
    "\nGrading rubric:\n"
    "- FAIL if the response states any specific fact that is NOT in the "
    "reference facts above, even if it sounds plausible or reasonable.\n"
    "- FAIL if the response does not address the question asked, or "
    "contradicts itself.\n"
    "- PASS if the response sticks to the reference facts above, or "
    "honestly says the information isn't available.\n"
    "Answer with exactly one word: PASS or FAIL."
)

One implementation detail worth calling out: when the judge’s reply cannot be cleanly parsed as PASS or FAIL, this code treats it as a FAIL, not a silent PASS:

text = r.json()["message"]["content"].strip().upper()
if "FAIL" in text:
    return "FAIL"
if "PASS" in text:
    return "PASS"
return "FAIL"  # unparseable output counts as a fail, not a silent pass

This is a fail-closed default on purpose. A judge that silently passes anything it cannot parse will quietly let bad output through; a judge that fails anything it cannot parse will, at worst, generate a false alarm you have to look at. In a grading pipeline, false alarms are far cheaper than silent misses.

Running both judge versions against the same sixteen real responses, at temperature 0, produced this real, unedited comparison:

      A     B     judge_v1  judge_v2
[10] FAIL  PASS    FAIL      FAIL
[11] FAIL  PASS    PASS      PASS
[12] FAIL  PASS    FAIL      FAIL
[13] FAIL  FAIL    FAIL      FAIL
[14] FAIL  FAIL    FAIL      PASS
[15] FAIL  FAIL    PASS      PASS

(Items 0 through 9 are omitted here; both judges correctly passed all ten in-knowledge-base answers, matching both human raters.)

Step 7: Score the Judge Against Both Humans

Now compute percent agreement and kappa for each judge against each rater, plus the human-to-human number from Step 4 as a reference ceiling:

=== Human-human ceiling ===
A vs B  percent=0.8125  kappa=0.5556

=== Judge v1 (minimal prompt, no rubric) ===
v1 vs A percent=0.8750  kappa=0.7143
v1 vs B percent=0.8125  kappa=0.4545

=== Judge v2 (reference facts + explicit rubric) ===
v2 vs A percent=0.8125  kappa=0.5556
v2 vs B percent=0.7500  kappa=0.1795

This is a genuinely useful, and slightly counter-intuitive, real result. Judge v1, with no rubric at all, actually agrees with Strict-Sam better than the two humans agree with each other (kappa 0.7143, “substantial,” versus the human ceiling of 0.5556, “moderate”). Judge v2, with the detailed rubric added, drops back down to exactly the human ceiling against Strict-Sam, and falls to 0.1795, “slight,” against Lenient-Lee.

Here is the lesson worth sitting with: adding the rubric did not make the judge objectively “better.” It made the judge agree more closely with one specific rater’s philosophy (Strict-Sam’s, since the rubric text is essentially Strict-Sam’s rules written out) at the direct expense of agreement with the other. There is no single number that tells you whether a judge is “good.” There is only how well it matches the judgment of the specific humans whose standards you actually care about. If your team’s real reviewers lean lenient, a strict rubric will make your judge look worse in the metric that matters, even while it looks the same or better on paper against an abstract standard.

Step 8: Is the v1-vs-v2 Difference Real, or Noise?

Judge v1 beat judge v2 by 0.1587 kappa points against Rater A (0.7143 minus 0.5556). On a set of only sixteen items, is that a real, repeatable difference, or could it just as easily have gone the other way with slightly different examples? This is exactly the question a statistical significance test answers, and it is the question most eval write-ups skip entirely.

The tool here is a paired percentile bootstrap, a resampling method introduced by Bradley Efron in 1979. The idea: repeatedly resample your sixteen items, with replacement, to build many “what if” versions of your golden set, and recompute both judges’ kappa against Rater A on each resampled version. If the gap between v1 and v2 holds up across almost all of those resamples, it is a real effect. If the gap regularly flips sign or collapses to zero, your original difference was likely just a lucky draw.

def paired_bootstrap_kappa_diff(items, ref_key, metric_a_key, metric_b_key,
                                 n_resamples=10000, seed=42):
    rng = random.Random(seed)
    n = len(items)
    deltas = []
    for _ in range(n_resamples):
        sample = [items[rng.randrange(n)] for _ in range(n)]
        ref = [row[ref_key] for row in sample]
        m_a = [row[metric_a_key] for row in sample]
        m_b = [row[metric_b_key] for row in sample]
        delta = cohens_kappa(m_a, ref) - cohens_kappa(m_b, ref)
        deltas.append(delta)
    return deltas

Each resample draws sixteen items with replacement from the original sixteen (so some items appear more than once and others not at all in a given resample) and recomputes the same paired difference on that resample. Running 10,000 resamples with a fixed seed, for reproducibility, and reading off the 2.5th and 97.5th percentiles as a 95 percent confidence interval:

>>> python bootstrap.py
Observed kappa(v1,A) - kappa(v2,A) = 0.1587
10,000-resample bootstrap 95% CI of the delta: [0.0000, 0.5143]
Two-sided bootstrap p-value: 0.7134
The interval contains zero: on n=16 items, we cannot distinguish
this gap from noise. Do not ship the rubric change on this evidence alone.

The confidence interval brushes right up against zero, and the p-value is 0.7134, nowhere close to a conventional significance threshold. In plain terms: with only sixteen examples, this dataset cannot tell you with any confidence whether judge v1 is really better than judge v2 against Strict-Sam, worse, or the same. The honest conclusion is not “v1 wins.” It is “we do not have enough data yet to know,” and the fix is a bigger golden set, not a stronger opinion.

Common Mistakes and Gotchas

• Reporting percent agreement alone. As Steps 3 through 5 showed twice, a high percent-agreement number can hide a judge, or a rater pair, that is barely better than chance once the label distribution is skewed.
• Treating one human rater as ground truth. If you only ever validate your judge against a single reviewer, you cannot tell calibration from shared bias. Use at least two independent raters and report agreement with each one, not a blended “average.”
• Ignoring sample size when comparing two kappa scores. A 0.16-point kappa gap looks decisive until you check whether it survives resampling. Small golden sets, sixteen items here, are common in early-stage evaluation work, and they make single point-estimate comparisons unreliable on their own.
• Forgetting that a rubric change invalidates old validation. Judge v2 was never separately re-validated as a concept; it was compared using the exact same human labels as v1. Any time you change a judge’s prompt, its kappa against your human raters needs to be recomputed, not assumed to carry over.
• Letting unparseable judge output default to PASS. A judge that fails open on malformed responses will silently let bad output through exactly when something has already gone wrong with the judge itself. Fail closed instead, as this tutorial’s judge_call does.

How to Verify It All Works End to End

The whole pipeline is covered by an automated test suite that checks the deterministic pieces (percent agreement, Cohen’s kappa, and the bootstrap function’s reproducibility) without needing a running model:

$ pytest test_agreement.py -v
test_agreement.py::test_percent_agreement_basic PASSED
test_agreement.py::test_kappa_perfect_agreement_is_one PASSED
test_agreement.py::test_kappa_below_percent_agreement_when_skewed PASSED
test_agreement.py::test_kappa_matches_hand_worked_example PASSED
test_agreement.py::test_confusion_matrix_counts PASSED
test_agreement.py::test_bootstrap_is_reproducible_with_fixed_seed PASSED
test_agreement.py::test_bootstrap_changes_with_different_seed PASSED

7 passed in 0.05s

To confirm the full, model-dependent pipeline works end to end on your own machine, run the scripts in order with Ollama running locally: generate_responses.py, then human_ratings.py, then judge.py, then metrics_report.py, then bootstrap.py. Each step writes its output into responses.json so the next step can read it, and every number printed along the way should match what is shown in this post if you are running the same model at temperature 0.

Next Steps

This tutorial used a simple Pass/Fail label and the standard, unweighted Cohen’s kappa. If your own ratings are ordinal, say a 1 to 5 quality scale, look into weighted kappa, which penalizes a 1-versus-5 disagreement more heavily than a 4-versus-5 disagreement. If you need more than two raters, or some raters skip some items, Krippendorff’s alpha generalizes the same underlying idea to those cases.

For the mechanics of building the judge itself in more depth, see How to Build a Local LLM-as-a-Judge Evaluation Harness for AI Agents. To validate retrieval quality specifically, rather than end-to-end response quality, see How to Evaluate RAG Retrieval Quality With Precision, Recall, and MRR in Python. And if the failure mode you care about is a judge or an agent being confidently wrong rather than honestly uncertain, How to Build an AI System That Knows When It Doesn’t Know With Ollama and Python covers that from a different angle.