How to Build a Semantic Cache for LLM Calls With Ollama and Python

If you have ever built a chatbot or an AI-powered support tool, you have probably noticed that the same question keeps arriving in a dozen different disguises. “What is the capital of France?” and “What’s France’s capital city?” mean exactly the same thing to a human, but to a computer they are two completely different strings of text. If your app calls a large language model (LLM) every single time, you pay for and wait on the same answer over and over, just because the wording changed.

In this tutorial you will build a semantic cache: a cache that recognizes when a new question means the same thing as one it has already answered, even when the words are different, and reuses the old answer instead of generating a new one. You will do this with Ollama running locally and plain Python, no cloud API keys and no paid services required. Along the way you will hit a real, dangerous failure mode of semantic caching (a cache that confidently returns the wrong answer) and learn how to guard against it.

Everything in this post was written by actually running the code against a local Ollama server. Every number, every log line, and every “gotcha” below is copied from a real terminal, not invented for effect.

What you will build, and why it matters

A cache, in the general programming sense, is a place to store the result of an expensive operation so you can reuse it instead of redoing the work. The classic web-development version of this is keying a dictionary (or a Redis store) by the exact input string: if the new request’s text matches a previous request’s text byte for byte, return the stored result.

That “exact match” approach works fine for something like an HTTP GET request to a fixed URL. It falls apart for LLM traffic, because real users (and real upstream systems) never ask the same question the same way twice. “What is the capital of France?”, “What’s France’s capital city?”, and “Can you tell me the capital city of France” are three different strings, so an exact-match cache treats them as three unrelated questions and pays for three full LLM generations, even though a human would answer all three identically.

A semantic cache fixes this by comparing meaning instead of text. It does this with embeddings: a way of converting a sentence into a list of numbers (a vector) such that sentences with similar meaning produce similar vectors. To decide whether two vectors are “similar,” you compute their cosine similarity, a single number between -1 and 1 that measures the angle between them, where 1 means “pointing in exactly the same direction” (same meaning) and 0 means “unrelated.” You do not need any prior machine-learning background to follow along; both of those concepts are explained and built from scratch below.

Prerequisites

• Ollama installed and running on Windows, macOS, or Linux. If you don’t have it yet, our guide to installing Ollama and running your first local chatbot covers that from scratch first.
• Python 3.10 or newer, with the requests package installed (pip install requests).
• About 400 MB of free disk space for a small embedding model, on top of whatever chat model you already have pulled.
• Basic Python: functions, dictionaries, and classes. No prior experience with embeddings, vectors, or machine learning is assumed; every formula is written out and explained.

This tutorial uses qwen2.5:1.5b as the chat model, a small model that runs comfortably on CPU. Any chat model already pulled on your machine will work the same way; just change CHAT_MODEL in the code below.

Step 1: Pull an embedding model

Ollama’s chat models (like qwen2.5:1.5b or llama3.2) are built to generate text, not to produce embeddings. For that you need a model specifically trained to output vectors. Pull one now:

ollama pull nomic-embed-text

This downloads a 274 MB model that “can only be used to generate embeddings,” according to its own model card. It has a 2K token context window, which is more than enough for typical questions. The pull itself is quick:

pulling manifest
pulling 970aa74c0a90: 100% |################| 274 MB
pulling c71d239df917: 100% |################|  11 KB
verifying sha256 digest
writing manifest
success

Step 2: Generate your first embedding

Ollama exposes embeddings through a REST endpoint, POST /api/embed, which takes a model name and an input string and returns a list of floating-point numbers. Create a small helper file, ollama_client.py:

import time
import requests

OLLAMA_URL = "http://127.0.0.1:11434"
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "qwen2.5:1.5b"


def embed(text: str) -> list[float]:
    resp = requests.post(
        f"{OLLAMA_URL}/api/embed",
        json={"model": EMBED_MODEL, "input": text},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["embeddings"][0]


def chat(prompt: str) -> tuple[str, float]:
    """Returns (answer_text, wall_clock_seconds)."""
    start = time.perf_counter()
    resp = requests.post(
        f"{OLLAMA_URL}/api/generate",
        json={
            "model": CHAT_MODEL,
            "prompt": prompt,
            "stream": False,
        },
        timeout=120,
    )
    resp.raise_for_status()
    elapsed = time.perf_counter() - start
    return resp.json()["response"].strip(), elapsed


def cosine_similarity(a: list[float], b: list[float]) -> float:
    if len(a) != len(b):
        raise ValueError(f"embedding length mismatch: {len(a)} vs {len(b)} (did the embedding model change?)")
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = sum(x * x for x in a) ** 0.5
    norm_b = sum(y * y for y in b) ** 0.5
    return dot / (norm_a * norm_b)

Notice the URL uses 127.0.0.1 rather than localhost. On some Windows machines, resolving the hostname localhost adds a large, consistent delay to every single request (multiple seconds), even though the server itself replies in milliseconds. We measured this directly: 10 requests to localhost averaged over 2 seconds each, and the identical 10 requests to 127.0.0.1 averaged about 0.1 seconds each. If your own local Ollama calls feel mysteriously slow, that hostname is worth a look; our benchmarking-LLM-inference tutorial digs into that specific quirk in more depth. For the rest of this post we use 127.0.0.1 so the timings below reflect real caching behavior, not a DNS quirk.

Now call it once:

from ollama_client import embed

vector = embed("What is the capital of France?")
print("Embedding length:", len(vector))
print("First 5 numbers:", [round(v, 4) for v in vector[:5]])

Embedding length: 768
First 5 numbers: [-0.0017, 0.0227, -0.1531, -0.0208, 0.0559]

That is the whole idea of an embedding: one sentence became a list of 768 numbers. Those numbers on their own mean nothing to a human, but their relationship to other sentences’ numbers is exactly what lets a computer recognize “same meaning, different words.”

Step 3: Measure meaning with cosine similarity

The cosine_similarity function above computes the cosine of the angle between two vectors: multiply the vectors’ matching components together and sum them (the dot product), then divide by the product of their lengths (their norms). The result stays between -1 and 1 no matter how long the original sentences are, which is what makes it useful for comparing meaning rather than just word overlap.

Let’s see what real similarity scores look like for a few sentence pairs, from an obvious paraphrase down to something totally unrelated:

from ollama_client import embed, cosine_similarity

pairs = [
    ("What is the capital of France?", "What's France's capital city?"),
    ("What is the capital of France?", "Can you tell me the capital city of France"),
    ("What is the capital of France?", "What is the capital of Germany?"),
    ("What is the capital of France?", "How do I bake sourdough bread?"),
]

for a, b in pairs:
    vec_a = embed(a)
    vec_b = embed(b)
    sim = cosine_similarity(vec_a, vec_b)
    print(f"{sim:.4f}  |  {a!r}  vs  {b!r}")

0.9641  |  'What is the capital of France?'  vs  "What's France's capital city?"
0.9415  |  'What is the capital of France?'  vs  'Can you tell me the capital city of France'
0.7674  |  'What is the capital of France?'  vs  'What is the capital of Germany?'
0.3398  |  'What is the capital of France?'  vs  'How do I bake sourdough bread?'

The two real paraphrases score above 0.94. The completely unrelated sourdough-bread question scores 0.34, comfortably far away. But look at the France-versus-Germany pair: 0.7674. That is a different question with a different correct answer, yet it scores a lot closer to the paraphrases than to the unrelated question. Keep that number in mind; it is about to cause a real bug.

Step 4: Watch a naive cache miss an obvious paraphrase

Before building the semantic version, it’s worth seeing the problem it solves. Here is the simplest possible cache, keyed by the exact prompt string:

from ollama_client import chat

naive_cache: dict[str, str] = {}


def naive_cached_chat(prompt: str) -> tuple[str, bool, float]:
    if prompt in naive_cache:
        return naive_cache[prompt], True, 0.0
    answer, elapsed = chat(prompt)
    naive_cache[prompt] = answer
    return answer, False, elapsed


queries = [
    "What is the capital of France?",
    "What's France's capital city?",
    "What is the capital of France?",  # exact repeat
]

for q in queries:
    answer, hit, elapsed = naive_cached_chat(q)
    print(f"hit={hit}  time={elapsed:.2f}s  query={q!r}")
    print(f"    answer: {answer}")

hit=False  time=0.49s  query='What is the capital of France?'
    answer: The capital of France is Paris.
hit=False  time=0.35s  query="What's France's capital city?"
    answer: France's capital city is Paris.
hit=True  time=0.00s  query='What is the capital of France?'
    answer: The capital of France is Paris.

The paraphrase in the second line pays full price again, even though it needed the exact same answer as the first line. Only the byte-for-byte repeat on the third line hits the cache. This is the gap a semantic cache is meant to close.

Step 5: Build the semantic cache

The idea: for each new prompt, embed it, compare it against the embeddings of every prompt already in the cache, and if the closest one scores above some threshold, reuse its stored answer instead of calling the model again. Save this as semantic_cache.py:

import time
from dataclasses import dataclass, field

from ollama_client import chat, cosine_similarity, embed


@dataclass
class CacheEntry:
    prompt: str
    embedding: list[float]
    answer: str
    created_at: float = field(default_factory=time.time)


class SemanticCache:
    def __init__(self, threshold: float = 0.90, ttl_seconds: float | None = None):
        self.threshold = threshold
        self.ttl_seconds = ttl_seconds
        self.entries: list[CacheEntry] = []
        self.hits = 0
        self.misses = 0

    def _best_match(self, query_embedding: list[float]) -> tuple[CacheEntry | None, float]:
        best_entry = None
        best_score = -1.0
        for entry in self.entries:
            score = cosine_similarity(query_embedding, entry.embedding)
            if score > best_score:
                best_score = score
                best_entry = entry
        return best_entry, best_score

    def get(self, prompt: str) -> tuple[str, bool, float, float]:
        """Returns (answer, was_cache_hit, similarity_score, wall_clock_seconds)."""
        start = time.perf_counter()
        query_embedding = embed(prompt)
        best_entry, best_score = self._best_match(query_embedding)

        if best_entry is not None and best_score >= self.threshold:
            self.hits += 1
            elapsed = time.perf_counter() - start
            return best_entry.answer, True, best_score, elapsed

        answer, _ = chat(prompt)
        self.entries.append(CacheEntry(prompt=prompt, embedding=query_embedding, answer=answer))
        self.misses += 1
        elapsed = time.perf_counter() - start
        return answer, False, best_score, elapsed

This first version is intentionally simple (a linear scan over every stored embedding), which is plenty fast for a cache of a few thousand entries and easy to reason about. We’ll harden it with a TTL and a size cap in Step 10.

Step 6: See real savings across paraphrased questions

from semantic_cache import SemanticCache

cache = SemanticCache(threshold=0.90)

queries = [
    "What is the capital of France?",
    "What's France's capital city?",
    "Can you tell me the capital city of France",
    "Tell me France's capital.",
]

for q in queries:
    answer, hit, score, elapsed = cache.get(q)
    print(f"hit={hit}  similarity={score:.4f}  time={elapsed:.2f}s  query={q!r}")
    print(f"    answer: {answer[:80]}")

print(f"\nTotals: {cache.hits} hits, {cache.misses} misses")

hit=False  similarity=-1.0000  time=0.50s  query='What is the capital of France?'
    answer: The capital of France is Paris.
hit=True  similarity=0.9641  time=0.06s  query="What's France's capital city?"
    answer: The capital of France is Paris.
hit=True  similarity=0.9415  time=0.05s  query='Can you tell me the capital city of France'
    answer: The capital of France is Paris.
hit=True  similarity=0.9373  time=0.05s  query="Tell me France's capital."
    answer: The capital of France is Paris.

Totals: 3 hits, 1 misses

The first call pays the full generation cost (0.50s). Every rephrasing after that returns in about 0.05 to 0.06 seconds, roughly an 8 to 10 times speedup, because it only had to embed the new question and compare vectors, never running the chat model again. This is the entire value proposition of a semantic cache: real savings on real paraphrases, with no exact-string matching required.

Step 7: The gotcha, a “safe-looking” threshold that serves a wrong answer

Here is where semantic caching gets dangerous if you are not careful. A threshold of 0.90 felt strict in Step 6. But recall the France-versus-Germany pair from Step 3 scored 0.7674, and suppose a slightly less cautious threshold of 0.75 is chosen instead, because it “still looks high”:

from semantic_cache import SemanticCache

# A threshold that looks "reasonably strict" at first glance.
cache = SemanticCache(threshold=0.75)

answer1, hit1, score1, _ = cache.get("What is the capital of France?")
print(f"hit={hit1}  similarity={score1:.4f}  query='What is the capital of France?'")
print(f"    answer: {answer1}")

answer2, hit2, score2, _ = cache.get("What is the capital of Germany?")
print(f"hit={hit2}  similarity={score2:.4f}  query='What is the capital of Germany?'")
print(f"    answer: {answer2}")

if hit2:
    print("\n*** WRONG: served the France cache entry for a Germany question ***")

hit=False  similarity=-1.0000  query='What is the capital of France?'
    answer: The capital of France is Paris.
hit=True  similarity=0.7674  query='What is the capital of Germany?'
    answer: The capital of France is Paris.

*** WRONG: served the France cache entry for a Germany question ***

That is a genuinely wrong answer, delivered with total confidence and zero indication anything went wrong. This is the single biggest risk of semantic caching: unlike a slow cache miss, a wrong cache hit looks exactly like a correct answer to both your code and your user. A cache that is merely slow is an inconvenience. A cache that is confidently wrong is a trust problem.

Step 8: Why raising the threshold alone doesn’t fix it

The instinctive fix is “just raise the threshold higher.” Before doing that, it’s worth actually measuring where real paraphrases and real near-miss questions land, instead of guessing. Here is a small labeled test set: five genuine paraphrase pairs (should cache-hit) and five “hard negative” pairs, questions that share almost the same sentence structure but need a different answer (should not cache-hit):

from ollama_client import embed, cosine_similarity

# label, query_a, query_b
pairs = [
    ("MATCH", "What is the capital of France?", "What's France's capital city?"),
    ("MATCH", "How do I reset my password?", "What are the steps to reset my password?"),
    ("MATCH", "What's the weather like today?", "Can you tell me today's weather?"),
    ("MATCH", "How much RAM do I need to run a 7B model?", "What's the minimum RAM for running a 7 billion parameter model?"),
    ("MATCH", "Is Python installed on this machine?", "Do I have Python installed?"),
    ("HARD_NEGATIVE", "What is the capital of France?", "What is the capital of Germany?"),
    ("HARD_NEGATIVE", "How do I reset my password?", "How do I reset my email?"),
    ("HARD_NEGATIVE", "What's the weather like today?", "What was the weather like yesterday?"),
    ("HARD_NEGATIVE", "How much RAM do I need to run a 7B model?", "How much RAM do I need to run a 70B model?"),
    ("HARD_NEGATIVE", "Is Python installed on this machine?", "Is Docker installed on this machine?"),
]

results = [(label, cosine_similarity(embed(a), embed(b)), a, b) for label, a, b in pairs]
results.sort(key=lambda r: -r[1])
for label, sim, a, b in results:
    print(f"{sim:.4f}  {label:14s}  {a!r} <-> {b!r}")

0.9641  MATCH           'What is the capital of France?' <-> "What's France's capital city?"
0.9541  MATCH           'How do I reset my password?' <-> 'What are the steps to reset my password?'
0.9072  HARD_NEGATIVE   "What's the weather like today?" <-> 'What was the weather like yesterday?'
0.8841  MATCH           'Is Python installed on this machine?' <-> 'Do I have Python installed?'
0.8783  MATCH           "What's the weather like today?" <-> "Can you tell me today's weather?"
0.8331  HARD_NEGATIVE   'How much RAM do I need to run a 7B model?' <-> 'How much RAM do I need to run a 70B model?'
0.7993  MATCH           'How much RAM do I need to run a 7B model?' <-> "What's the minimum RAM for running a 7 billion parameter model?"
0.7850  HARD_NEGATIVE   'How do I reset my password?' <-> 'How do I reset my email?'
0.7674  HARD_NEGATIVE   'What is the capital of France?' <-> 'What is the capital of Germany?'
0.6834  HARD_NEGATIVE   'Is Python installed on this machine?' <-> 'Is Docker installed on this machine?'

Look closely at the ordering. The real matches range from 0.7993 to 0.9641. The hard negatives range from 0.6834 to 0.9072. Those two ranges overlap: the “what was the weather like yesterday” hard negative (0.9072) scores higher than the legitimate “7B model RAM” paraphrase (0.7993). There is no single threshold value that perfectly separates every real match from every dangerous near-miss in this test set. Push the threshold up to avoid the Germany mistake, and you also start rejecting real paraphrases as fresh, more expensive cache misses; push it down to catch more paraphrases, and some hard negatives sneak through as wrong-answer cache hits. A single similarity number is a useful signal, but on its own it is not a safe gate.

Step 9: Add a second safety layer: a distinguishing-token guard

Since a pure similarity score cannot be trusted alone, add a second, independent check: does either sentence contain a specific fact (a number, a proper noun, or a word like “today” or “yesterday”) that the other sentence contradicts? These are exactly the tokens most likely to flip the correct answer. Save this as guard.py:

import re

RELATIVE_TIME_WORDS = {"today", "yesterday", "tomorrow", "now", "currently", "yet"}


def distinguishing_tokens(text: str) -> set[str]:
    """Numbers, mid-sentence proper nouns, and relative-time words: the
    tokens most likely to flip the correct answer if they change."""
    words = re.findall(r"[A-Za-z0-9]+", text)
    tokens = set()
    for i, word in enumerate(words):
        lowered = word.lower()
        if lowered in RELATIVE_TIME_WORDS:
            tokens.add(lowered)
        elif re.search(r"\d", word):
            digits = re.findall(r"\d+", word)
            tokens.update(digits)
        elif word[0].isupper() and i > 0:  # skip sentence-initial capital
            tokens.add(lowered)
    return tokens


def safe_to_reuse(cached_prompt: str, new_prompt: str) -> bool:
    """False if both prompts carry distinguishing tokens and those tokens
    disagree. True (allow) when there's nothing to disagree about."""
    cached_tokens = distinguishing_tokens(cached_prompt)
    new_tokens = distinguishing_tokens(new_prompt)
    if not cached_tokens or not new_tokens:
        return True
    return bool(cached_tokens & new_tokens)

The rule in safe_to_reuse is deliberately conservative: if neither sentence has anything worth flagging (no numbers, no proper nouns, no relative-time words), allow the reuse. If either does, only allow it when the two sets of tokens actually overlap. “What is the capital of France?” and “What’s France’s capital city?” both mention France, so they overlap and the guard allows it. “What is the capital of France?” and “What is the capital of Germany?” mention different countries and don’t overlap, so the guard blocks it, independent of whatever the similarity score says.

A bug I found while testing the guard

The first version of this function used the regex [A-Za-z0-9']+ (including the apostrophe) to split words, so it could handle contractions cleanly. Testing it against the labeled pairs above turned up a real bug: it classified the genuine “capital of France” paraphrase pair as unsafe, when it should have been allowed. The cause was a tokenization mismatch: “France” in the first sentence became the token france, but “France‘s” in the second sentence, apostrophe included, became the token france's, a different string that never matched. Switching the regex to [A-Za-z0-9]+ (dropping the apostrophe) splits “France’s” into “France” and “s” instead, so the meaningful token normalizes to france on both sides and the two texts finally agree. The code above already has that fix; if you ever build a similar guard yourself and it starts rejecting things that clearly should match, check exactly this: how possessives and contractions are being tokenized.

Re-running the full labeled set with a threshold of 0.85 plus this guard:

THRESHOLD = 0.85
for label, a, b in pairs:
    sim = cosine_similarity(embed(a), embed(b))
    would_hit_threshold_only = sim >= THRESHOLD
    would_hit_with_guard = would_hit_threshold_only and safe_to_reuse(a, b)
    print(f"label={label:14s} sim={sim:.4f}  threshold_only_hit={would_hit_threshold_only!s:5s}  with_guard_hit={would_hit_with_guard!s:5s}")

MATCH          sim=0.9641  threshold_only_hit=True   with_guard_hit=True
MATCH          sim=0.9541  threshold_only_hit=True   with_guard_hit=True
MATCH          sim=0.8783  threshold_only_hit=True   with_guard_hit=True
MATCH          sim=0.7993  threshold_only_hit=False  with_guard_hit=False
MATCH          sim=0.8841  threshold_only_hit=True   with_guard_hit=True
HARD_NEGATIVE  sim=0.7674  threshold_only_hit=False  with_guard_hit=False
HARD_NEGATIVE  sim=0.7850  threshold_only_hit=False  with_guard_hit=False
HARD_NEGATIVE  sim=0.9072  threshold_only_hit=True   with_guard_hit=False
HARD_NEGATIVE  sim=0.8331  threshold_only_hit=False  with_guard_hit=False
HARD_NEGATIVE  sim=0.6834  threshold_only_hit=False  with_guard_hit=False

9/10 correctly classified with threshold=0.85 + guard

All five hard negatives are now correctly blocked, four of them by the threshold alone and the “weather today versus yesterday” one specifically by the guard (its 0.9072 similarity clears the threshold, but “today” and “yesterday” don’t overlap as distinguishing tokens). The one “wrong” row is the RAM paraphrase at 0.7993: it falls below the 0.85 threshold, so it becomes a cache miss and pays for a fresh generation. That is a real limitation, not a bug: this configuration trades away a little bit of hit rate on borderline phrasing in exchange for zero wrong-answer hits on every hard negative we tested. It is also worth being honest that the guard itself is a heuristic, not a proof: a hard negative that differs only in an ordinary noun (like “reset my password” versus “reset my email”, both lowercase, neither a number) would slip past the guard on similarity alone if it ever scored above the threshold. In our test set it didn’t (0.7850, well below 0.85), but a production system should not assume every such case will be so lucky; treat this as one layer of defense, not a guarantee.

Step 10: Confirm the fix, then add TTL and a size cap for production hygiene

Wire the guard into SemanticCache itself, add an optional time-to-live so stale answers eventually expire, and cap how many entries it will hold:

from guard import safe_to_reuse
from ollama_client import chat, cosine_similarity, embed

class SemanticCache:
    def __init__(
        self,
        threshold: float = 0.90,
        ttl_seconds: float | None = None,
        use_guard: bool = True,
        max_entries: int = 1000,
    ):
        self.threshold = threshold
        self.ttl_seconds = ttl_seconds
        self.use_guard = use_guard
        self.max_entries = max_entries
        self.entries: list[CacheEntry] = []
        self.hits = 0
        self.misses = 0
        self.guard_blocked = 0

    def _prune_expired(self) -> None:
        if self.ttl_seconds is None:
            return
        cutoff = time.time() - self.ttl_seconds
        self.entries = [e for e in self.entries if e.created_at >= cutoff]

    def get(self, prompt: str) -> tuple[str, bool, float, float]:
        self._prune_expired()
        start = time.perf_counter()
        query_embedding = embed(prompt)
        best_entry, best_score = self._best_match(query_embedding)

        above_threshold = best_entry is not None and best_score >= self.threshold
        if above_threshold and self.use_guard and not safe_to_reuse(best_entry.prompt, prompt):
            above_threshold = False
            self.guard_blocked += 1

        if above_threshold:
            self.hits += 1
            elapsed = time.perf_counter() - start
            return best_entry.answer, True, best_score, elapsed

        answer, _ = chat(prompt)
        self.entries.append(CacheEntry(prompt=prompt, embedding=query_embedding, answer=answer))
        if len(self.entries) > self.max_entries:
            self.entries.pop(0)  # evict oldest
        self.misses += 1
        elapsed = time.perf_counter() - start
        return answer, False, best_score, elapsed

(The _best_match method is unchanged from Step 5.) Now confirm the original Germany bug is actually fixed with this real configuration, and that genuine paraphrases still work:

cache = SemanticCache(threshold=0.85, use_guard=True)

a1, hit1, score1, _ = cache.get("What is the capital of France?")
a2, hit2, score2, _ = cache.get("What is the capital of Germany?")
assert hit2 is False, "guard should have blocked this"

a3, hit3, score3, _ = cache.get("What's France's capital city?")
assert hit3 is True, "a genuine paraphrase should still hit"

hit=False  similarity=-1.0000  query='What is the capital of France?'
    answer: The capital of France is Paris.
hit=False  similarity=0.7674  query='What is the capital of Germany?'
    answer: The capital of Germany is Berlin.

Fixed: the Germany question correctly triggered a fresh generation instead of reusing Paris.

hit=True  similarity=0.9641  query="What's France's capital city?"
    answer: The capital of France is Paris.

Germany now correctly gets its own, correct answer (“Berlin”), and the real paraphrase still hits the cache. Finally, confirm TTL expiry actually prunes stale entries instead of just tracking a number nobody checks:

import time
cache = SemanticCache(threshold=0.85, ttl_seconds=2)

cache.get("What is the capital of France?")
cache.get("What's France's capital city?")  # hits immediately, entry is fresh

print("sleeping 3s so the entry ages past the 2s TTL...")
time.sleep(3)

_, hit3, _, _ = cache.get("What's France's capital city?")
assert hit3 is False, "expired entry should not be reused"

hit=False  (first call, always a miss)
hit=True  (immediate paraphrase, still fresh)
sleeping 3s so the entry ages past the 2s TTL...
hit=False  (same paraphrase, but entry expired)

TTL expiry confirmed: the stale entry was pruned and a fresh answer was generated.

Step 11: Verify everything end-to-end with pytest

Before trusting any of this, write it down as tests you can rerun after every change. Save as test_semantic_cache.py:

import time
from semantic_cache import SemanticCache


def test_exact_repeat_hits():
    cache = SemanticCache(threshold=0.85)
    cache.get("What is the capital of France?")
    _, hit, score, _ = cache.get("What is the capital of France?")
    assert hit is True
    assert score > 0.99


def test_paraphrase_hits():
    cache = SemanticCache(threshold=0.85)
    cache.get("What is the capital of France?")
    answer, hit, _, _ = cache.get("What's France's capital city?")
    assert hit is True
    assert "Paris" in answer


def test_different_country_does_not_reuse_wrong_answer():
    cache = SemanticCache(threshold=0.85)
    cache.get("What is the capital of France?")
    answer, hit, _, _ = cache.get("What is the capital of Germany?")
    assert hit is False
    assert "Paris" not in answer


def test_unrelated_question_misses():
    cache = SemanticCache(threshold=0.85)
    cache.get("What is the capital of France?")
    _, hit, score, _ = cache.get("How do I bake sourdough bread?")
    assert hit is False
    assert score < 0.5


def test_ttl_expiry():
    cache = SemanticCache(threshold=0.85, ttl_seconds=1)
    cache.get("What is the capital of France?")
    time.sleep(1.5)
    _, hit, _, _ = cache.get("What's France's capital city?")
    assert hit is False


def test_max_entries_evicts_oldest():
    cache = SemanticCache(threshold=0.85, max_entries=2)
    cache.get("What is the capital of France?")
    cache.get("How do I bake sourdough bread?")
    cache.get("What is the weather like on Mars?")
    assert len(cache.entries) == 2
    assert cache.entries[0].prompt != "What is the capital of France?"

$ pytest test_semantic_cache.py -v
test_semantic_cache.py::test_exact_repeat_hits PASSED
test_semantic_cache.py::test_paraphrase_hits PASSED
test_semantic_cache.py::test_different_country_does_not_reuse_wrong_answer PASSED
test_semantic_cache.py::test_unrelated_question_misses PASSED
test_semantic_cache.py::test_ttl_expiry PASSED
test_semantic_cache.py::test_max_entries_evicts_oldest PASSED

6 passed in 27.94s

Common mistakes and gotchas

Trusting a single threshold without measuring your own data. As Step 8 showed, "match" and "hard negative" similarity ranges can overlap. A threshold that looks safe by eyeballing one or two examples can still let real mistakes through. Build a small labeled set of your own real traffic patterns (or at least your app's most common question categories) and check the actual score distribution before picking a number.

Mixing embeddings from two different models. A vector from nomic-embed-text and a vector from any other embedding model are not comparable, even if they happen to be the same length; the numbers encode meaning according to that specific model's training, not some universal scale. If you ever change your embedding model, you must re-embed everything already in the cache, not just new entries. Our cosine_similarity function above raises a clear ValueError if the two vectors' lengths don't even match, which is a cheap first line of defense, but it cannot catch the more dangerous case of two same-length vectors from different models producing a technically valid, meaningless similarity score.

Caching personalized or per-user answers globally. If your LLM calls include per-user context ("what's on my calendar today"), a semantic cache shared across users will happily serve one person's calendar to another person who asked a similarly worded question. Scope the cache per user (or per tenant) whenever the answer depends on who is asking, the same principle covered in our per-user token isolation tutorial.

No invalidation path for a bad or outdated cached answer. A wrong answer, once cached, stays wrong for as long as its TTL allows (or forever, if you set no TTL at all). Always cap the cache with a TTL that matches how quickly the underlying facts can change, and keep a way to manually evict a specific bad entry once you find one.

Assuming a "high" similarity score is inherently safe. As Step 3 showed, 0.7674 sounds high in isolation, but it was still a wrong-answer risk. Only comparative measurement against your own real match and near-miss pairs tells you whether a given number is actually safe for your use case.

How to verify your own semantic cache actually works

Before shipping a semantic cache in front of real traffic, confirm at minimum:

1. An exact repeat of a previous question hits the cache with a similarity at or near 1.0.
2. A genuine paraphrase of a previous question also hits the cache, and returns the same stored answer.
3. A "hard negative", a question that shares vocabulary or structure with a cached question but needs a different answer, does not hit the cache. Build this test set from your own domain, not just the examples in this post.
4. An unrelated question always misses and triggers a fresh generation.
5. Entries actually expire once their TTL passes, and the cache actually stops growing once it reaches its size cap.

If you cannot check all five with real, reproducible tests (not just a spot check by hand), the cache is not ready for real traffic.

Next steps

This tutorial built a semantic cache from scratch, on purpose, so the mechanics and the failure modes are fully visible. Once you understand what is happening under the hood, a few natural directions to keep learning:

• Swap the linear-scan lookup for a real vector index (like pgvector or a dedicated vector database) once you have far more cache entries than a simple loop can compare quickly; our vector database latency piece is a good next read on what that lookup actually costs at scale.
• Look at existing semantic-caching libraries, like GPTCache, to see how a mature project handles the same trade-offs discussed in Step 8, and compare their default thresholds against your own measurements.
• For borderline similarity scores that the threshold and guard can't confidently resolve either way, consider a third layer: asking the LLM itself a short, cheap "are these two questions asking for the same information?" verification prompt, similar in spirit to our structured-output validation and retries tutorial.
• If your cache is meant to reduce cost against a hosted, metered API rather than a local Ollama model, revisit our prompt cache hit rate tutorial, which covers a different but related kind of caching: the model provider's own prefix-based prompt cache, as opposed to the application-level semantic cache built here.