How to Measure Whether Cross-Encoder Reranking Actually Improves RAG Retrieval

A recent Red Hat post on optimizing enterprise RAG (retrieval-augmented generation) pipelines makes a specific claim: pairing precise, smaller chunks with a cross-encoder reranker consistently delivers more optimal cost profiles than either technique alone. That is a reasonable claim, and it matches what a lot of production RAG advice says: retrieve broadly, then rerank precisely.

This tutorial does not ask you to take that claim on faith. You will build a working cross-encoder reranker from scratch, wire it into a small RAG retrieval pipeline, and then measure, with a real eval set and real numbers, whether it actually improves results. The honest answer you will find by the end is more interesting than a simple yes: reranking works well in some cases, makes things worse in others, and cannot fix every kind of retrieval mistake. Understanding exactly when it helps and when it does not is the actual skill this tutorial teaches.

What You Will Build, and Why This Problem Is Real

If you have built a RAG pipeline before (if not, see this site’s local RAG Q&A agent tutorial first), you already know the basic shape: turn a user’s question into a vector using an embedding model, compare that vector against precomputed vectors for every chunk of your documents using cosine similarity, and hand the top few matches to a language model as context.

This works well for obviously different topics. It works much worse when two chunks are semantically close but factually different: a “Free tier rate limit” paragraph and a “Pro tier rate limit” paragraph use almost identical vocabulary and sentence structure. An embedding model, which compresses each chunk into a single fixed-size vector without ever seeing the user’s exact question next to the exact chunk text, can genuinely struggle to tell them apart. This is the retrieval-stage version of a classic problem: the model that is fast enough to search millions of documents is not always precise enough to get the top result right.

The proposed fix is a reranker: a second model that looks at the user’s question and a candidate chunk together, and outputs a single relevance score for that specific pairing. Because it re-scores a short list of candidates rather than searching a whole corpus, it can afford to be much slower and more careful than the first-stage retrieval. The most common implementation is a cross-encoder, and that is what you will build here.

By the end of this tutorial you will have:

• A small, deliberately confusable knowledge base and a 20-question eval set with known-correct answers
• An embedding-only retrieval pipeline using Ollama, and a measurement of where it gets things wrong
• A cross-encoder reranking stage added on top of it, using a real pretrained model
• A rigorous, repeatable measurement of whether reranking actually improved accuracy on this data (it is not a clean win, and you will see exactly why)
• A demonstration of the one failure mode reranking can never fix
• A deterministic pytest suite for the pipeline logic that does not require a live model or network access to run

Prerequisites

• Python 3.10 or newer (built and tested here on Python 3.13.14)
• Ollama installed and running locally, with the nomic-embed-text embedding model pulled (ollama pull nomic-embed-text)
• About 1.2GB of free disk space the first time you set this up: sentence-transformers pulls in PyTorch as its backend (roughly 540MB by itself) plus the Hugging Face transformers library (roughly 115MB), and the cross-encoder model itself is about 90MB, downloaded once and cached outside your virtual environment
• Comfort with basic Python and the general shape of a RAG pipeline; no prior reranking experience assumed

Install the Python dependencies in a fresh virtual environment:

python -m venv venv
venv\Scripts\activate          # on Windows
# source venv/bin/activate     # on macOS/Linux

pip install sentence-transformers ollama pytest numpy

This tutorial was built and verified against sentence-transformers 6.1.0, ollama 0.6.2, pytest 9.1.1, and numpy 2.5.3.

Step 1: Build a Deliberately Confusable Knowledge Base

Real internal documentation is full of near-duplicate sections: one paragraph per pricing tier, one paragraph per error code, one paragraph per API version. That repetition is exactly what makes retrieval hard, so the example knowledge base below is built the same way on purpose: a fictional API platform’s docs, with three separate paragraphs about rate limits (Free, Pro, Enterprise) that share most of their vocabulary and differ only in the specific number and audience.

Save this as knowledge_base.py:

"""A small internal API documentation knowledge base for a fictional company, Vectra."""

DOCS = {
    "auth-bearer": "Vectra API requests must include an Authorization header with a Bearer token obtained from the /oauth/token endpoint. Tokens expire after 1 hour.",
    "auth-apikey": "API keys for the Vectra API are created in the dashboard under Settings > API Keys and never expire unless revoked.",
    "auth-scopes": "OAuth tokens can be scoped to read-only or read-write access when requested from the /oauth/token endpoint.",
    "rate-free": "Free tier accounts are limited to 100 requests per minute per API key. Requests beyond this return HTTP 429.",
    "rate-pro": "Pro tier accounts are limited to 1,000 requests per minute per API key. Requests beyond this return HTTP 429.",
    "rate-enterprise": "Enterprise tier accounts negotiate a custom requests-per-minute ceiling with their account manager, typically 10,000 or higher.",
    "rate-burst": "All tiers allow a short burst of up to 3x their steady-state rate limit for a maximum of 5 seconds before throttling kicks in.",
    "webhook-retry": "Webhook deliveries are retried up to 5 times with exponential backoff starting at 30 seconds.",
    "webhook-sig": "Every webhook payload is signed with an HMAC-SHA256 signature in the X-Vectra-Signature header so you can verify it came from Vectra.",
    "webhook-events": "Vectra sends webhook events for order.created, order.updated, and order.cancelled. Other event types are not currently supported.",
    "pagination-default": "List endpoints are paginated with a default page size of 25, controlled by the limit query parameter.",
    "pagination-max": "The maximum page size for any list endpoint is 100. Requesting a larger limit silently caps it at 100.",
    "versioning": "Vectra API versions are specified via the Accept header. Deprecated versions are supported for 12 months after a new version ships.",
    "errors-429": "A 429 Too Many Requests response includes a Retry-After header telling you how many seconds to wait before retrying.",
    "errors-401": "A 401 Unauthorized response means your Bearer token is missing, expired, or malformed. Re-authenticate via /oauth/token.",
    "errors-500": "A 500 Internal Server Error means something failed on Vectra's side. These are automatically retried by our SDKs.",
    "idempotency": "POST requests accept an optional Idempotency-Key header. Replaying the same key within 24 hours returns the original response.",
    "sandbox": "The sandbox environment at sandbox.vectra.dev mirrors production but resets all data every 24 hours and never charges real payment methods.",
    "sla": "Vectra's SLA guarantees 99.95% uptime for Pro and Enterprise tiers, measured monthly, excluding scheduled maintenance windows.",
    "deprecation": "Deprecated endpoints return a Sunset header with the exact date they will stop working, at least 90 days in advance.",
}

# One realistic, paraphrased user question per doc, with its known-correct doc id.
EVAL_SET = {
    "auth-bearer": "How long is my access token good for before I need a new one?",
    "auth-apikey": "Where in the dashboard do I go to create a new credential?",
    "auth-scopes": "Can I request a token that's only allowed to read data, not write it?",
    "rate-free": "I'm not paying for anything, how many calls can I make per minute?",
    "rate-pro": "How many API calls per minute am I allowed on the Pro plan?",
    "rate-enterprise": "We need way more than a thousand requests a minute, who do we talk to?",
    "rate-burst": "Can I temporarily exceed my rate limit for a short spike in traffic?",
    "webhook-retry": "What happens if a webhook delivery fails the first time?",
    "webhook-sig": "How do I confirm a webhook actually came from Vectra and not an attacker?",
    "webhook-events": "Which events will trigger a webhook to fire?",
    "pagination-default": "If I don't specify a limit, how many results come back on a list call?",
    "pagination-max": "What's the largest page size I can request from a list endpoint?",
    "versioning": "Can I control what shape the response is in based on which version I request?",
    "errors-429": "What header tells me how long to wait after getting rate limited?",
    "errors-401": "My requests are failing with an unauthorized error, what does that mean?",
    "errors-500": "Do I need to manually retry if the server itself has an internal failure?",
    "idempotency": "How do I safely retry a POST without creating a duplicate?",
    "sandbox": "Is there a way to test my integration without spending real money?",
    "sla": "What uptime does Vectra promise for paying customers?",
    "deprecation": "How far in advance do you warn before killing off old endpoints?",
}

Notice that EVAL_SET maps each question to exactly one correct document id. This is the ground truth you will measure both pipelines against. Without it, you would be judging retrieval quality by eyeballing a few results, which is exactly how teams end up shipping a reranker (or a chunking change, or a new embedding model) that they believe helped, without ever actually checking. The RAG evaluation tutorial on this site covers precision, recall, and MRR in more depth if you want the full methodology; here you will compute a simpler top-1 accuracy and MRR@5 directly against this eval set.

Verify this step worked

Run python -c "from knowledge_base import DOCS, EVAL_SET; print(len(DOCS), len(EVAL_SET))". You should see 20 20. If Python cannot import the module, make sure knowledge_base.py is in your current directory.

Step 2: Retrieve Candidates With Embeddings (the Bi-Encoder Stage)

The first stage of any RAG pipeline uses what is called a bi-encoder: a model that encodes the query and each document independently into fixed-size vectors. Because the document vectors never depend on the query, you can compute them once, store them, and reuse them for every future search. That is what makes this stage fast enough to run over millions of documents.

Save this as retrieval.py. It uses Ollama’s nomic-embed-text model, the same embedding model used elsewhere on this site, so if you already have it pulled you do not need anything new:

"""Embedding-based (bi-encoder) retrieval: the fast, first-stage search."""
import ollama
import numpy as np

_client = ollama.Client()


def embed(text: str) -> np.ndarray:
    response = _client.embed(model="nomic-embed-text", input=text)
    return np.array(response["embeddings"][0])


def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))


def build_index(docs: dict[str, str]) -> dict[str, np.ndarray]:
    """Embed every document once, up front. This is the part that scales:
    embedding N documents costs O(N), and each embedding can be cached and
    reused for every future query."""
    return {doc_id: embed(text) for doc_id, text in docs.items()}


def retrieve(query: str, index: dict[str, np.ndarray], k: int = 5) -> list[tuple[float, str]]:
    """Return the top-k (score, doc_id) pairs by cosine similarity, sorted
    highest-first. This costs O(N) comparisons per query against precomputed
    vectors, cheap even for large N."""
    query_vector = embed(query)
    scored = [(cosine(query_vector, vector), doc_id) for doc_id, vector in index.items()]
    scored.sort(reverse=True)
    return scored[:k]

Try it on one question first, to see the shape of the output before running the full eval set:

from knowledge_base import DOCS
from retrieval import build_index, retrieve

index = build_index(DOCS)
for score, doc_id in retrieve("How do I safely retry a POST without creating a duplicate?", index, k=5):
    print(f"{score:.4f}  {doc_id}")

The real captured output from this exact query is:

0.6553  errors-429
0.6380  idempotency
0.5365  webhook-retry
0.5343  errors-401
0.4735  errors-500

Look closely: the correct answer, idempotency, is ranked second, not first. errors-429 narrowly wins because its text mentions “retrying” too, in the context of rate-limit backoff, not duplicate-safe POST requests. Cosine similarity has no way to know that these are different kinds of retrying; it only sees that the two chunks use overlapping vocabulary in a similar structure. This is a genuine, reproducible retrieval mistake, not a contrived one, and it is exactly the kind of error reranking is supposed to fix.

Verify this step worked

You should see five rows printed with scores between roughly 0.4 and 0.7. If Ollama is not running, you will get a connection error; start it and confirm ollama list shows nomic-embed-text before continuing.

Step 3: What a Cross-Encoder Reranker Actually Is

A cross-encoder is architecturally different from the bi-encoder you just used. Instead of encoding the query and a document into two separate vectors and comparing them afterward, it feeds the query and the document into the model together, at the same time, so the model’s attention layers can directly compare specific words in the query against specific words in the document. Sentence-transformers’ own documentation puts the distinction plainly: with a bi-encoder, “we pass to a BERT independently the sentences A and B, which result in the sentence embeddings u and v,” while “for a Cross-Encoder, we pass both sentences simultaneously to the Transformer network. It produces then an output value… indicating the similarity.”

That joint attention is what makes cross-encoders more precise: they can weigh the fact that your question said “Pro” specifically, right next to a candidate answer that also says “Pro,” in a way two independently-computed vectors cannot. But it comes at a real cost. Because the model needs both texts present to produce a score, “a Cross-Encoder does not produce a sentence embedding,” and you cannot precompute anything. The sentence-transformers documentation quantifies exactly how much this costs at scale: “Clustering 10,000 sentence with CrossEncoders would require computing similarity scores for about 50 Million sentence combinations, which takes about 65 hours. With a Bi-Encoder, you compute the embedding for each sentence, which takes only 5 seconds.” That is why the standard pattern is to use a bi-encoder to narrow a huge corpus down to a short list of candidates first, then use a cross-encoder only on that short list.

The specific model you will use, cross-encoder/ms-marco-MiniLM-L6-v2, was trained on the MS MARCO Passage Retrieval dataset, described in sentence-transformers’ pretrained cross-encoder model list as “a large dataset with real user queries from Bing search engine with annotated relevant text passages.” Keep that origin in mind: this model learned what “relevant” looks like from real Bing search queries, not from API documentation. That fact turns out to matter a lot in Step 5.

Step 4: Add a Reranking Step

Save this as reranker.py:

"""Cross-encoder reranking: the slow, precise second stage."""
from sentence_transformers import CrossEncoder

_model = None


def get_model() -> CrossEncoder:
    global _model
    if _model is None:
        _model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
    return _model


def rerank(query: str, candidates: list[tuple[float, str]], docs: dict[str, str]) -> list[tuple[float, str]]:
    """Score every candidate jointly with the query and return a new
    ranking. Unlike retrieve(), this cannot be precomputed: the model has
    to see the query and the candidate text together."""
    model = get_model()
    doc_ids = [doc_id for _, doc_id in candidates]
    pairs = [[query, docs[doc_id]] for doc_id in doc_ids]
    scores = model.predict(pairs)
    reranked = sorted(zip(scores, doc_ids), key=lambda pair: pair[0], reverse=True)
    return [(float(score), doc_id) for score, doc_id in reranked]

Notice the shape of the input to model.predict(): a list of [query, passage] pairs, not a list of texts. This is the single most common mistake when first using a CrossEncoder: it has no .encode() method that returns a reusable vector the way SentenceTransformer does. If you pass it a flat list of strings instead of paired lists, it will raise an error rather than silently doing the wrong thing, which is a small mercy.

Run the same idempotency question from Step 2 back through retrieve() and then through rerank():

from knowledge_base import DOCS
from retrieval import build_index, retrieve
from reranker import rerank

index = build_index(DOCS)
query = "How do I safely retry a POST without creating a duplicate?"
candidates = retrieve(query, index, k=5)
reranked = rerank(query, candidates, DOCS)
for score, doc_id in reranked:
    print(f"{score:8.4f}  {doc_id}")

The real captured output is:

-7.4928  errors-429
-7.8400  idempotency
-10.9607  webhook-retry
-11.0260  errors-401
-11.1765  errors-500

The reranker did not fix this one. It still ranks errors-429 first, by a narrow margin over the correct idempotency answer (negative scores are normal here; sentence-transformers’ own documentation notes these raw values “can reasonably range between -10 and 10” rather than the 0-to-1 range you would get with a probability, so what matters is their relative order for a given query, not their absolute sign). This is a genuinely important, easy-to-miss lesson: adding a reranker is not a guarantee that any specific mistake gets corrected. Whether it does depends on whether the reranker’s own training gives it a better signal for that particular query than the embedding model had, and here it did not.

Verify this step worked

The first time you run this, sentence-transformers will download the model (about 90MB) from Hugging Face; you will see a progress bar and a one-time warning about unauthenticated requests, which is safe to ignore for a small public model like this one. Confirm you see five scores printed, all negative, in descending order.

Step 5: Stop Trusting One Example, Measure Both Pipelines

One query is an anecdote, not evidence. To find out whether reranking actually helps on average, you need to run both pipelines against every question in EVAL_SET and compare top-1 accuracy and MRR@5 (mean reciprocal rank within the top 5 results; 1.0 if the correct answer is ranked first, 0.5 if second, 0 if it never appears in the top 5 at all).

Save this as run_eval.py:

"""Measure embedding-only retrieval against embedding-plus-rerank on the same
20-question eval set, and report whether reranking actually helped."""
from knowledge_base import DOCS, EVAL_SET
from retrieval import build_index, retrieve
from reranker import rerank

CANDIDATE_WINDOW = 5


def reciprocal_rank(ranked_ids: list[str], correct_id: str) -> float:
    if correct_id not in ranked_ids:
        return 0.0
    return 1.0 / (ranked_ids.index(correct_id) + 1)


def main() -> None:
    index = build_index(DOCS)

    embed_hits = 0
    rerank_hits = 0
    embed_rr_total = 0.0
    rerank_rr_total = 0.0
    fixed, broke, still_wrong = [], [], []

    print(f"{'expected':<18} {'embed_top1':<18} {'rerank_top1':<18} outcome")
    print("=" * 72)

    for correct_id, query in EVAL_SET.items():
        candidates = retrieve(query, index, k=CANDIDATE_WINDOW)
        embed_ranked_ids = [doc_id for _, doc_id in candidates]
        embed_top1 = embed_ranked_ids[0]

        reranked = rerank(query, candidates, DOCS)
        rerank_ranked_ids = [doc_id for _, doc_id in reranked]
        rerank_top1 = rerank_ranked_ids[0]

        embed_ok = embed_top1 == correct_id
        rerank_ok = rerank_top1 == correct_id
        embed_hits += embed_ok
        rerank_hits += rerank_ok
        embed_rr_total += reciprocal_rank(embed_ranked_ids, correct_id)
        rerank_rr_total += reciprocal_rank(rerank_ranked_ids, correct_id)

        if not embed_ok and rerank_ok:
            outcome = "FIXED"
            fixed.append((correct_id, query))
        elif embed_ok and not rerank_ok:
            outcome = "BROKE"
            broke.append((correct_id, query))
        elif not embed_ok and not rerank_ok:
            outcome = "STILL_WRONG"
            still_wrong.append((correct_id, query))
        else:
            outcome = "both_ok"

        print(f"{correct_id:<18} {embed_top1:<18} {rerank_top1:<18} {outcome}")

    n = len(EVAL_SET)
    print("\n=== Summary ===")
    print(f"Top-1 accuracy   embedding-only: {embed_hits}/{n} ({embed_hits/n:.0%})")
    print(f"Top-1 accuracy   embedding+rerank: {rerank_hits}/{n} ({rerank_hits/n:.0%})")
    print(f"MRR@{CANDIDATE_WINDOW}          embedding-only: {embed_rr_total/n:.4f}")
    print(f"MRR@{CANDIDATE_WINDOW}          embedding+rerank: {rerank_rr_total/n:.4f}")
    print(f"\nFIXED by reranking:       {len(fixed)}  {fixed}")
    print(f"BROKEN by reranking:      {len(broke)}  {broke}")
    print(f"STILL_WRONG after rerank: {len(still_wrong)}  {still_wrong}")


if __name__ == "__main__":
    main()

Run python run_eval.py. The real captured output, unedited, is:

expected           embed_top1         rerank_top1        outcome
========================================================================
auth-bearer        auth-bearer        auth-bearer        both_ok
auth-apikey        auth-apikey        auth-apikey        both_ok
auth-scopes        auth-scopes        auth-scopes        both_ok
rate-free          rate-free          rate-pro            BROKE
rate-pro           rate-pro           rate-pro           both_ok
rate-enterprise    rate-pro           rate-pro           STILL_WRONG
rate-burst         rate-burst         rate-burst         both_ok
webhook-retry      webhook-retry      webhook-retry      both_ok
webhook-sig        webhook-sig        webhook-sig        both_ok
webhook-events     webhook-events     webhook-events     both_ok
pagination-default pagination-default pagination-default both_ok
pagination-max     pagination-max     pagination-max     both_ok
versioning         versioning         idempotency        BROKE
errors-429         errors-429         errors-429         both_ok
errors-401         errors-401         errors-401         both_ok
errors-500         errors-500         errors-500         both_ok
idempotency        errors-429         errors-429         STILL_WRONG
sandbox            sandbox            sandbox            both_ok
sla                sla                sla                both_ok
deprecation        deprecation        deprecation        both_ok

=== Summary ===
Top-1 accuracy   embedding-only: 18/20 (90%)
Top-1 accuracy   embedding+rerank: 16/20 (80%)
MRR@5          embedding-only: 0.9375
MRR@5          embedding+rerank: 0.8792

FIXED by reranking:       0  []
BROKEN by reranking:      2  [('rate-free', "I'm not paying for anything, how many calls can I make per minute?"), ('versioning', 'Can I control what shape the response is in based on which version I request?')]
STILL_WRONG after rerank: 2  [('rate-enterprise', 'We need way more than a thousand requests a minute, who do we talk to?'), ('idempotency', 'How do I safely retry a POST without creating a duplicate?')]

Read that summary carefully, because it contradicts the simple version of the reranking story. On this eval set, adding the reranker did not fix a single mistake (FIXED: 0), broke two answers that embedding-only retrieval already had right, and left two genuine misses uncorrected. Both top-1 accuracy (90% down to 80%) and MRR@5 (0.9375 down to 0.8792) got measurably worse after adding a reranking stage that, on paper, should have helped.

This is not a trick result. It is what you get when you plug a general-purpose, off-the-shelf reranker into a narrow technical domain it was never tuned for, and then actually measure the outcome instead of assuming the textbook description of the technique applies unchanged. Let's look at exactly why, using the two broken cases.

Why the reranker broke two correct answers

Run this to see the full candidate list, with both scores, for the two BROKE cases:

from knowledge_base import DOCS
from retrieval import build_index, retrieve
from reranker import rerank

CASES = {
    "rate-free": "I'm not paying for anything, how many calls can I make per minute?",
    "versioning": "Can I control what shape the response is in based on which version I request?",
}

index = build_index(DOCS)
for correct_id, query in CASES.items():
    print(f"\n=== {correct_id}: {query!r} ===")
    candidates = retrieve(query, index, k=5)
    reranked = rerank(query, candidates, DOCS)
    embed_scores = dict((doc_id, score) for score, doc_id in candidates)
    for ce_score, doc_id in reranked:
        marker = " [correct]" if doc_id == correct_id else ""
        print(f"{doc_id:<18} embed={embed_scores[doc_id]:<8.4f} rerank={ce_score:<9.4f}{marker}")

Captured output:

=== rate-free: "I'm not paying for anything, how many calls can I make per minute?" ===
rate-pro           embed=0.5703   rerank=-6.2766
rate-free          embed=0.5858   rerank=-6.3296   [correct]
rate-enterprise    embed=0.5761   rerank=-6.6846
errors-429         embed=0.5446   rerank=-10.8251
rate-burst         embed=0.5142   rerank=-11.1202

=== versioning: 'Can I control what shape the response is in based on which version I request?' ===
idempotency        embed=0.5437   rerank=-5.2371
errors-429         embed=0.5331   rerank=-6.1485
auth-scopes        embed=0.4572   rerank=-10.1654
versioning         embed=0.5507   rerank=-10.5780  [correct]
webhook-events      embed=0.5115   rerank=-11.0630 

Two different failure shapes are visible here. In the rate-free case, the cross-encoder is not confident about any of the three tier documents (all three score between -6.28 and -6.68, a gap of about 0.4 on a scale where clearly irrelevant documents score below -10). It is effectively guessing among near-ties, and it guesses wrong, while the embedding model's gap (0.5858 versus 0.5703) was small but in the correct direction. The user's phrase "not paying for anything" never uses the word "free," and the cross-encoder, trained on real search queries, appears to lean on literal wording more than the embedding model does here.

The versioning case is more dramatic: the reranker confidently ranks the correct document fourth out of five, well behind idempotency, even though embeddings correctly ranked it first. Nothing in the query text overlaps strongly with "Accept header" or "deprecated," so the cross-encoder has little literal signal to work with, and its semantic judgment on this narrow API-documentation domain is simply wrong here.

Step 6: Where Reranking Actually Earns Its Keep

Before you conclude that this specific reranker is useless, look at what happens when a query does share literal wording with the correct document. Save and run this:

"""Where reranking earns its keep: a query that says the tier name explicitly,
so the cross-encoder has clean lexical and semantic signal to work with."""
from reranker import get_model

query = "What is the rate limit for the Pro tier?"
candidates = {
    "rate-pro": "Pro tier accounts are limited to 1,000 requests per minute per API key.",
    "rate-free": "Free tier accounts are limited to 100 requests per minute per API key.",
    "webhook-retry": "Webhook deliveries are retried up to 5 times with exponential backoff.",
}

model = get_model()
pairs = [[query, text] for text in candidates.values()]
scores = model.predict(pairs)

for (doc_id, text), score in sorted(zip(candidates.items(), scores), key=lambda pair: pair[1], reverse=True):
    print(f"{score:8.4f}  {doc_id:14s} {text}")

Captured output:

  0.8948  rate-pro       Pro tier accounts are limited to 1,000 requests per minute per API key.
 -8.3614  rate-free      Free tier accounts are limited to 100 requests per minute per API key.
-11.4187  webhook-retry  Webhook deliveries are retried up to 5 times with exponential backoff.

This is a completely different picture: a wide, confident, correct separation between the right tier document (0.8948, a strongly positive score) and the two wrong candidates (-8.3614 and -11.4187). This is what cross-encoder reranking is genuinely good at: when the query gives it clean literal and semantic signal to work with, it can distinguish near-duplicate documents far more confidently than a bi-encoder's single-vector comparison.

Put the two findings together and the real lesson emerges: reranking with an off-the-shelf, general-purpose cross-encoder is not unconditionally good or bad, it is query-and-domain dependent. This particular model was trained on MS MARCO, real Bing search queries against web passages, not on internal API documentation. It performs best on queries that resemble what it was trained on, and its judgment on this narrow, jargon-heavy domain is not automatically better than a general-purpose embedding model's. Production teams that fine-tune their own reranker on labeled query-document pairs from their actual domain, or that A/B test several open reranker models against a real eval set like the one you just built, routinely see different results than teams that bolt on a popular pretrained reranker and assume it helps because a blog post said so.

Step 7: The One Failure Reranking Can Never Fix

There is a structural limit to what reranking can do, no matter how good the reranker is: it can only reorder the candidates it is given. If the first-stage retrieval buries the correct document so far down that it never makes it into the candidate window, no amount of reranking that window will find it.

Here is a real example from this exact knowledge base. Save and run this:

"""What reranking can't do: recover a document the retrieval step buried so
deep it never makes it into the candidate window."""
import time
from knowledge_base import DOCS
from retrieval import build_index, retrieve, cosine, embed
from reranker import rerank, get_model

QUERY = "If I upgrade past the free plan, does my rate limit go up automatically?"
CORRECT = "rate-pro"

index = build_index(DOCS)

query_vector = embed(QUERY)
full_ranking = sorted(((cosine(query_vector, v), doc_id) for doc_id, v in index.items()), reverse=True)
ranked_ids = [doc_id for _, doc_id in full_ranking]
print(f"Embedding-only rank of the correct doc ({CORRECT}): {ranked_ids.index(CORRECT) + 1} of {len(DOCS)}")

top5 = retrieve(QUERY, index, k=5)
reranked_top5 = rerank(QUERY, top5, DOCS)
print(f"Reranked top-1 from the top-5 window: {reranked_top5[0][1]}")
print(f"Correct doc was even in that window: {CORRECT in [d for _, d in top5]}")

model = get_model()
t0 = time.perf_counter()
all_pairs = [[QUERY, DOCS[doc_id]] for doc_id in DOCS]
full_scores = model.predict(all_pairs)
full_elapsed = time.perf_counter() - t0
full_reranked_ids = [doc_id for _, doc_id in sorted(zip(full_scores, list(DOCS.keys())), reverse=True)]
print(f"\nFull-corpus rerank ({len(DOCS)} docs) took {full_elapsed:.4f}s")
print(f"Full-corpus rerank rank of {CORRECT}: {full_reranked_ids.index(CORRECT) + 1} of {len(DOCS)}")

t1 = time.perf_counter()
window_pairs = [[QUERY, DOCS[doc_id]] for _, doc_id in top5]
model.predict(window_pairs)
window_elapsed = time.perf_counter() - t1
print(f"Top-5-window rerank took {window_elapsed:.4f}s ({full_elapsed / window_elapsed:.1f}x faster than full corpus)")

Captured output:

Embedding-only rank of the correct doc (rate-pro): 15 of 20
Reranked top-1 from the top-5 window: rate-burst
Correct doc was even in that window: False

Full-corpus rerank (20 docs) took 0.0371s
Full-corpus rerank rank of rate-pro: 11 of 20
Top-5-window rerank took 0.0110s (3.4x faster than full corpus)

Two things are worth sitting with here. First, the embedding step ranked the actually-correct document 15th out of 20 for this question, so reranking the top-5 window was never going to find it: it was never in the window to begin with. Second, and more surprising, even reranking the entire 20-document corpus, bypassing the candidate window altogether, only moved the correct document up to 11th place, still nowhere near the top. The cross-encoder itself does not judge this document as highly relevant to this particular question, because the question asks about an implicit consequence ("does my rate limit go up automatically" when upgrading) that no single chunk states directly; it requires combining two separate facts. Neither the embedding model nor the reranker is built to do that kind of multi-hop inference from one short passage.

The practical lesson: reranking is not a substitute for good retrieval. It is a refinement on top of it. If your retrieval stage has genuinely bad recall for a class of questions, no reranker downstream will fix it, and widening the candidate window only helps if the reranker actually agrees the missing document is relevant once it sees it, which is not guaranteed.

The latency numbers also explain why nobody reranks an entire large corpus per query: this toy 20-document example was already 3.4x slower to rerank in full than to rerank just a 5-document window, and that gap grows linearly with corpus size. A production knowledge base with thousands of chunks would make full-corpus reranking completely impractical per query, which is exactly the 65-hours-versus-5-seconds tradeoff described in Step 3.

Common Mistakes and Gotchas

• Passing plain strings to CrossEncoder.predict(). It expects a list of [query, passage] pairs, not a flat list of texts. A CrossEncoder also cannot produce a reusable embedding for a single piece of text the way SentenceTransformer can; it only ever scores a specific pair.
• Reranking a whole corpus instead of a retrieved candidate window. As Step 7 showed, this scales badly (O(N) cross-encoder calls per query instead of O(N) once, up front, for embeddings) and still is not guaranteed to find the right answer if the query needs inference the model was not trained to do.
• Trusting a vendor claim or blog post that reranking "consistently" improves results, without measuring it on your own data. That is the whole finding of this tutorial: a real, popular, pretrained cross-encoder measurably hurt accuracy on this specific 20-question, jargon-heavy eval set, while clearly helping on a differently-phrased query about the same documents. The only way to know which situation you are in is to build an eval set like EVAL_SET above and measure both pipelines, the same way you just did.
• Treating a negative cross-encoder score as "not relevant." The raw scores from ms-marco-MiniLM-L6-v2 are logits, not probabilities, and sentence-transformers documents them as reasonably ranging from -10 to 10 (this tutorial's own scores went slightly past that at -11.4187, which is a normal reminder that "reasonably" is not a hard limit); what matters is a score's relative order among candidates for the same query, not whether it is above or below zero.

Step 8: A Deterministic Test Suite for the Pipeline Logic

The scripts above are demos: they need Ollama running and download a model the first time. Your actual application code, the windowing, sorting, and MRR-calculation logic, should be tested independently of that, so your test suite runs in milliseconds in CI without needing a live model or network access. Save this as test_reranker.py:

"""Deterministic tests for the retrieval/rerank logic. These use fake
embeddings and a fake cross-encoder so the suite runs in milliseconds and
doesn't need Ollama or a downloaded model."""
import numpy as np
import pytest

import retrieval
import reranker
from run_eval import reciprocal_rank


def test_cosine_identical_vectors_is_one():
    v = np.array([1.0, 2.0, 3.0])
    assert retrieval.cosine(v, v) == pytest.approx(1.0)


def test_cosine_orthogonal_vectors_is_zero():
    a = np.array([1.0, 0.0])
    b = np.array([0.0, 1.0])
    assert retrieval.cosine(a, b) == pytest.approx(0.0)


def test_retrieve_returns_top_k_sorted_descending(monkeypatch):
    fake_vectors = {
        "a": np.array([1.0, 0.0]),
        "b": np.array([0.9, 0.1]),
        "c": np.array([0.0, 1.0]),
    }
    # The query is identical to doc "a", so cosine similarity should rank
    # a > b > c.
    monkeypatch.setattr(retrieval, "embed", lambda text: np.array([1.0, 0.0]))

    results = retrieval.retrieve("anything", fake_vectors, k=2)

    assert [doc_id for _, doc_id in results] == ["a", "b"]
    assert results[0][0] > results[1][0]


def test_rerank_reorders_by_cross_encoder_score(monkeypatch):
    class FakeCrossEncoder:
        def predict(self, pairs):
            # Deliberately disagree with the input order: the LAST pair
            # should score highest, to prove rerank() actually re-sorts
            # rather than just passing candidates through.
            return list(range(len(pairs)))

    monkeypatch.setattr(reranker, "_model", FakeCrossEncoder())

    docs = {"x": "text x", "y": "text y", "z": "text z"}
    candidates = [(0.9, "x"), (0.5, "y"), (0.1, "z")]  # embedding order: x, y, z

    result = reranker.rerank("query", candidates, docs)

    assert [doc_id for _, doc_id in result] == ["z", "y", "x"]


def test_reciprocal_rank_of_top_result_is_one():
    assert reciprocal_rank(["a", "b", "c"], "a") == pytest.approx(1.0)


def test_reciprocal_rank_of_third_result():
    assert reciprocal_rank(["a", "b", "c"], "c") == pytest.approx(1.0 / 3.0)


def test_reciprocal_rank_zero_when_outside_window():
    # This is the recall-ceiling case: the correct doc never made it into
    # the candidate list at all, so its reciprocal rank is 0, not an error.
    assert reciprocal_rank(["a", "b", "c"], "not-present") == 0.0

Run pytest test_reranker.py -v. Real captured output:

test_reranker.py::test_cosine_identical_vectors_is_one PASSED            [ 14%]
test_reranker.py::test_cosine_orthogonal_vectors_is_zero PASSED          [ 28%]
test_reranker.py::test_retrieve_returns_top_k_sorted_descending PASSED   [ 42%]
test_reranker.py::test_rerank_reorders_by_cross_encoder_score PASSED     [ 57%]
test_reranker.py::test_reciprocal_rank_of_top_result_is_one PASSED       [ 71%]
test_reranker.py::test_reciprocal_rank_of_third_result PASSED            [ 85%]
test_reranker.py::test_reciprocal_rank_zero_when_outside_window PASSED   [100%]

7 passed in 5.73s

Notice the monkeypatch.setattr(reranker, "_model", FakeCrossEncoder()) line in the second test. It replaces the real cross-encoder with a fake object whose predict() method deliberately returns scores in reverse order from the input, specifically so the test fails loudly if rerank() ever stops actually sorting and just passes candidates through unchanged. That is a stronger test than one that merely checks the output looks reasonable; it checks that the sorting logic is doing real work.

How to Verify Everything Works End to End

1. Confirm Ollama is running and has nomic-embed-text: ollama list.
2. Run python run_eval.py. You should see the same 20-row table and summary statistics shown in Step 5 (top-1 accuracy 18/20 for embeddings, 16/20 for embedding-plus-rerank; these numbers are fully deterministic since both models are non-random).
3. Run pytest test_reranker.py -v. All 7 tests should pass in a few seconds, and this should work even with Ollama stopped, since these tests never call it.
4. Run the Step 6 confusable-pair script and confirm the Pro-tier document scores clearly positive (around 0.89) while the other two candidates score clearly negative (below -8).

If any of the live-model numbers differ slightly on your machine, check that you are using the exact same model names (nomic-embed-text and cross-encoder/ms-marco-MiniLM-L6-v2); different model versions can shift scores even when the overall pattern holds.

Next Steps

You now have a working reranking pipeline and, more importantly, a repeatable way to measure whether reranking (or any other RAG change) is actually helping. A few natural directions from here:

• Try a different pretrained cross-encoder, such as one of the larger ms-marco models or BAAI/bge-reranker-base, and rerun run_eval.py unchanged against it. Compare the accuracy and MRR numbers directly instead of guessing which model is "better" from its name or benchmark leaderboard position.
• If you have real user queries and click or thumbs-up/down data, that is exactly the kind of labeled pair data needed to fine-tune a cross-encoder on your own domain, which is the more reliable way to get the Step 6 behavior consistently rather than only on literally-phrased queries.
• Revisit the chunk-size and top-k tuning tutorial on this site and combine it with this one: sweep chunk size, top-k, and reranker on-or-off together in one evaluation harness, the same way a real AutoRAG-style optimization system would, rather than tuning one variable in isolation.
• Read the RAG evaluation methodology tutorial for a deeper treatment of precision, recall, and MRR beyond the simplified top-1/MRR@5 metrics used here.