How to Diagnose and Fix a Low Prompt Cache Hit Rate With Ollama

When you send the same instructions to a large language model over and over, a chatbot’s system prompt, a coding agent’s tool definitions, a RAG pipeline’s retrieved context, the model provider can often skip reprocessing the parts that have not changed and only pay for what is new. This is called prompt caching (also called prefix caching), and it is one of the highest-leverage levers for cutting the cost and latency of a production LLM system.

The catch is that it fails silently. A prompt that is technically correct and produces the right answer can still get a 0 percent cache hit rate because of one small ordering mistake, and nothing in the response tells you that happened unless you know where to look.

In this tutorial you will learn what prompt caching actually is, how to measure whether it is working, and how to fix the single most common mistake that kills it: putting a piece of dynamic content (a timestamp, a session ID, a user’s name) too early in the prompt. You will do this hands-on and for free, using a local model in Ollama, so you can see the effect on your own machine before you touch a paid API. Then you will carry the same diagnostic habit over to a hosted API (Anthropic’s Claude API is used as the concrete example), where the mechanism is explicit and the savings are real money, not just wall clock time.

Prerequisites

• A Linux machine. This tutorial was written and tested on Ubuntu with systemd; the Ollama commands work the same on macOS and WSL. Only the optional server-log inspection in Step 3 needs systemd’s journalctl and does not apply outside Linux (an alternative for other platforms is included).
• Ollama installed. This tutorial uses Ollama 0.31.2, but any recent release exposes the same API fields.
• At least one model pulled, of any size or family. This tutorial uses qwen3.5:4b (a 4.7 billion parameter Qwen 3.5 model, Q4_K_M quantization) because it is small enough to run on a CPU-only machine in a reasonable time. Anything from ollama pull llama3.2 to a 70B model works identically for the purposes of this tutorial; only the speed changes.
• Python 3.10 or newer with the requests library installed (pip install requests).
• Basic comfort with the terminal. No prior experience with LLM APIs is assumed; every field this tutorial reads from a response is explained the first time it appears.

Step 6 references a hosted API for context, but no API key is required to complete the hands-on portion of this tutorial (Steps 1 through 5).

What Prompt Caching Actually Is

Before touching a keyboard, it helps to know there are three different things people call “caching” when talking about LLMs, and mixing them up is the most common source of confusion.

KV cache: automatic, inside one request

When a model generates a reply, it computes an internal representation (the “key” and “value” tensors, hence “KV cache”) for every token it has seen so far, and reuses that representation for each new token it generates rather than recomputing everything from scratch. This happens automatically, for every request, on every model, with no configuration. You do not control it and you do not need to think about it. It is not what this tutorial is about.

Prompt cache (also called prefix cache): across requests, and the subject of this tutorial

This is the mechanism that lets a second request skip reprocessing content it already processed on a first request, as long as that content is an exact, byte-identical match at the front of the prompt. This is cross-request, it depends on you structuring your prompt correctly, and it is what breaks silently when done wrong.

Semantic cache: a different tool entirely

Some applications also use a “semantic cache”: a separate lookup layer, usually backed by embeddings and a vector store, that recognizes two differently worded questions as similar enough to reuse a previous answer, skipping inference entirely. That is an application-level cache you build yourself. It is unrelated to prompt caching and is not covered in this tutorial.

The one rule everything else follows from

Prompt caching keys on the exact prefix of your prompt: the tools, then the system prompt, then the message history, in that order. If even one byte differs anywhere in that prefix, everything after the differing byte has to be reprocessed. This single fact is the whole tutorial. Once you understand it, the rest is just learning to see it happen.

Step 1: Confirm Your Tools Are Ready

Check that Ollama is installed and that you have a model pulled:

ollama --version
ollama list

Expected output looks like this (your version and model list will differ):

ollama version is 0.31.2
NAME          ID              SIZE      MODIFIED
qwen3.5:4b    2a654d98e6fb    3.4 GB    6 days ago

If you see “command not found,” install Ollama first from ollama.com; installation itself is out of scope for this tutorial. If ollama list shows no models, pull one now:

ollama pull qwen3.5:4b

Confirm the Ollama background service is reachable over its local HTTP API:

curl -s http://localhost:11434/api/tags

You should get back a JSON blob listing your models. If you get “connection refused,” start the service with ollama serve in a separate terminal (on Linux with systemd it usually starts automatically; the command above simply confirms it is listening).

Finally, confirm Python has the requests library:

python3 -c "import requests; print(requests.__version__)"

If that raises ModuleNotFoundError, install it with pip install requests.

Step 2: Measure a Cold, Uncached Request

Ollama’s /api/chat endpoint returns timing fields in every response that tell you exactly how much work it did. The two fields you care about for this tutorial are:

• prompt_eval_count: the total size of the prompt on this request, in tokens
• prompt_eval_duration: how long the prompt-processing step took, in nanoseconds

If caching is working, the server’s own processing log (Step 3) will show it evaluating far fewer tokens than the prompt’s full length, because only the new part gets processed. If caching is not working, the server processes the full prompt length every time, no matter how much of that prompt it has already seen before.

Create a test script:

# cache_test.py
import requests, time

URL = "http://localhost:11434/api/chat"
MODEL = "qwen3.5:4b"

STABLE_SYSTEM = """You are a customer support assistant for a cloud storage company called Vaultbox.
Follow these rules on every reply:
1. Always be concise and professional.
2. Never invent pricing information; only use the pricing table below.
3. If you don't know an answer, say so and offer to escalate to a human agent.
4. Never ask for the customer's full password, only account email.

Vaultbox pricing table:
- Free tier: 5GB storage, 1 device
- Personal: $4.99/month, 200GB storage, 3 devices
- Family: $9.99/month, 2TB storage, 6 devices
- Business: $19.99/user/month, unlimited storage, unlimited devices, admin console, audit logs

Common troubleshooting steps for sync issues:
1. Check the customer's internet connection.
2. Confirm the Vaultbox client is on the latest version.
3. Ask them to sign out and back in.
4. Check for a paused sync state in the system tray icon.
5. If none of that works, ask them to send a diagnostic log bundle from Settings > Help > Export Logs.

Always close by asking if there is anything else you can help with.
"""

def chat(messages, label):
    t0 = time.time()
    resp = requests.post(URL, json={
        "model": MODEL,
        "messages": messages,
        "stream": False,
        "think": False,
        "options": {"temperature": 0, "num_predict": 60},
    }, timeout=280)
    wall = time.time() - t0
    data = resp.json()
    print(f"--- {label} ---")
    print(f"prompt_eval_count: {data.get('prompt_eval_count')}")
    print(f"prompt_eval_duration_ms: {data.get('prompt_eval_duration', 0)/1e6:.1f}")
    print(f"wall_clock_s: {wall:.2f}")
    print()
    return data

r1_messages = [
    {"role": "system", "content": STABLE_SYSTEM},
    {"role": "user", "content": "My files stopped syncing this morning, what should I check first?"},
]
r1 = chat(r1_messages, "REQUEST 1 (fresh conversation)")

Run it:

python3 cache_test.py

On the machine this tutorial was tested on, the output was:

--- REQUEST 1 (fresh conversation) ---
prompt_eval_count: 297
prompt_eval_duration_ms: 9905.7
wall_clock_s: 15.17

Your numbers will differ (CPU speed, model size, and prompt length all matter), but the shape will be similar: a fresh conversation with no prior context processes every token of the prompt (297 of them here), and that takes a noticeable amount of wall clock time, about 33 milliseconds per token on this all-CPU test machine.

Why disable thinking? Qwen 3.5 and several other current models support an extended “thinking” step before answering. It is valuable for hard reasoning tasks, but it makes replies much slower and is unrelated to what this tutorial measures, so the "think": False option turns it off to keep the test fast and focused. Leave it on if you want to see the model’s full reasoning; the caching behavior itself does not change, only the wall clock time does.

Step 3: Continue the Conversation and Watch the Cache Hit

Now append a follow-up turn to the exact same conversation, the way any normal multi-turn chat would, and send it again. Add this to the bottom of cache_test.py:

r2_messages = r1_messages + [
    {"role": "assistant", "content": r1["message"]["content"]},
    {"role": "user", "content": "That worked, thanks. One more thing, how much storage does the Family plan include?"},
]
r2 = chat(r2_messages, "REQUEST 2 (same conversation continues)")

Run the script again. Here is the real output from the test machine:

--- REQUEST 2 (same conversation continues) ---
prompt_eval_count: 385
prompt_eval_duration_ms: 3206.5
wall_clock_s: 5.51

Look at that first number. The full prompt for request 2 is 385 tokens long: the entire system prompt, the first question, the assistant’s first reply, and the second question, all combined. But prompt_eval_count reports the total context size, not the freshly processed count, so check the more precise number from the server’s own log, which breaks this out explicitly. If you have systemd, run this in another terminal right after sending the request:

journalctl -u ollama --since "2 minutes ago" | grep "prompt eval time"

The real log line from this test, matched to request 2, reads:

prompt eval time = 3206.50 ms / 92 tokens (34.85 ms per token, 28.69 tokens per second)

Ninety-two tokens, not 385. The stable system prompt plus the first question-and-answer pair (293 tokens worth) were restored from a cached checkpoint and never touched the model at all; only the 92 new tokens (the assistant’s first reply plus the new question) were actually evaluated. At roughly 33 to 35 milliseconds per token either way, reprocessing all 385 tokens fresh would have taken about 13.5 seconds. This request took 3.2 seconds. That gap, roughly a 4x speedup on the prompt-processing portion of this one request, is what a cache hit looks like, and it is the entire value proposition of prompt caching.

If you have systemd, you can see the exact moment this happens. Search the log for the word “checkpoint”:

journalctl -u ollama --since "2 minutes ago" | grep -i checkpoint

You should see a line like:

restored context checkpoint (pos_min = 292, pos_max = 292, n_tokens = 293, ...)

That is Ollama’s own llama.cpp-based server telling you, in its own words, that it found a matching checkpoint from request 1 and reused it instead of recomputing those 293 tokens. This is not a guess or an inference from timing; it is the server’s own log confirming the mechanism. If you are on macOS or do not have systemd, run ollama serve directly in a terminal instead of as a background service, and watch its standard output for the same lines.

Step 4: Break the Cache on Purpose

Now for the mistake this whole tutorial exists to help you avoid. Many real systems put something dynamic, a request ID, a timestamp, a session token, at or near the very start of the prompt, often inside the system prompt itself, treating it as harmless metadata. Reproduce that mistake on purpose. Add this to cache_test.py:

broken_system = f"[session_id={int(time.time())}]\n" + STABLE_SYSTEM
r3_messages = [
    {"role": "system", "content": broken_system},
    {"role": "user", "content": "My files stopped syncing this morning, what should I check first?"},
]
r3 = chat(r3_messages, "REQUEST 3 (dynamic field placed before the stable content)")

This starts a brand new conversation, not a continuation of requests 1 or 2, but its system prompt is almost identical: the same stable instructions and pricing table, with one extra line glued onto the very front. Run it. The real output:

--- REQUEST 3 (dynamic field placed before the stable content) ---
prompt_eval_count: 313
prompt_eval_duration_ms: 10444.3
wall_clock_s: 16.97

Compare this to request 1: a very similar total size (313 tokens versus 297), but the same full-reprocessing cost, about 33 milliseconds per token, exactly as if nothing had ever been cached. The server log confirms exactly what happened and why:

checking checkpoint with [292, 292] against 3...
forcing full prompt re-processing due to lack of cache data

The checkpoint check compares the new prompt against what is cached, byte by byte, from the start. Because the first line of this system prompt is a session ID that has never been seen before, the match breaks after only 3 tokens, and the entire rest of the prompt, all 313 tokens of it, has to be reprocessed from scratch. It does not matter that most of the prompt is identical to something already cached: a single differing byte at the front poisons everything that comes after it.

Fix it: move the dynamic content to the end

The fix is not to remove the dynamic field, it is to move it. Stable, reusable content goes first; anything that changes on every request goes last, after everything cacheable:

fixed_system = STABLE_SYSTEM + f"\n[session_id={int(time.time())}]"

If you use fixed_system as a continuation of a conversation whose stable prefix is already cached, rather than as a brand-new, unrelated conversation, the stable portion is restored from the checkpoint exactly as in Step 3, and only the short new suffix and the new question are freshly evaluated. The general principle, not specific to Ollama, is the one to remember: within a single prompt, order content from most stable to least stable, and put anything that changes on every single request as close to the very end as you can.

A gotcha: local caching is per-conversation, not global

If you experiment further and start a third, completely separate conversation that happens to share the same stable system prompt text as requests 1 and 2, do not be surprised if it is just as slow as request 3 was. Check how many parallel processing slots your local Ollama server is running with:

ps aux | grep llama-server

Look for a -np flag in the command line. A value of -np 1 means the server keeps exactly one cached checkpoint at a time, for whichever conversation it processed most recently. A third, unrelated conversation evicts the cache that requests 1 and 2 were sharing, even if its own text is similar. This is worth knowing for local testing, and it is also exactly why hosted, multi-tenant APIs cannot rely on “just continue the same conversation” and need an explicit, deliberately designed cache mechanism instead, which is what Step 6 covers.

Step 5: Verify the Whole Thing End to End

Before moving on, confirm you can reproduce the core result on your own machine:

1. Run the script from Step 2 through Step 4 in order: fresh conversation, continuation, broken prefix.
2. Confirm the continuation request’s server log shows a much smaller prompt eval time = ... tokens figure than the request’s total prompt length.
3. Confirm the broken-prefix request shows full reprocessing at roughly the same milliseconds-per-token rate as request 1, and that its log contains forcing full prompt re-processing due to lack of cache data.

If your numbers show that pattern, you have confirmed, on your own hardware, exactly how prefix-based caching decides what to keep and what to throw away. That understanding transfers directly to production hosted APIs, which is the next step.

Step 6: Apply the Same Diagnostic to a Hosted API

Local Ollama testing is free and a great way to build intuition, but production systems usually call a hosted API, where caching is billed explicitly rather than being an incidental side effect of one server process. Anthropic’s Claude API is a useful concrete example because its official prompt caching documentation states the mechanics precisely.

The cache_control parameter

You mark the end of a stable, reusable block of your prompt with a cache_control field. Render order matters here too: tools, then system prompt, then messages, in that order, exactly like the local test above.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": STABLE_SYSTEM,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "How much storage does the Family plan include?"}],
)

The cache_control: {"type": "ephemeral"} marker on that system block tells the API that everything up to and including this block is a candidate for reuse on a future request, for a limited time (5 minutes by default, or 1 hour if you set "ttl": "1h"). You are allowed up to 4 such markers in one request.

How much a cache actually costs and saves

Per Anthropic’s published pricing structure, a cache write costs 1.25 times the normal input price with the default 5-minute time to live (2 times with the 1-hour time to live), and a cache read costs about 0.1 times the normal input price. That means:

• With the 5-minute time to live, 2 requests sharing a prefix already come out ahead of paying full price twice (1.25x plus 0.1x is less than 2x).
• With the 1-hour time to live, you need at least 3 requests to come out ahead (2x plus 0.2x is less than 3x), because the write premium is higher.

The minimum prefix length is not the same for every model

A cache marker on a prefix shorter than a certain length silently does nothing: there is no error, the request just runs at full price. The minimum varies by model:

A 3,000-token system prompt caches fine on Sonnet 4.5 but silently will not on Opus 4.8. If you switch models and your cache savings disappear, check this table before assuming your code is broken.

Verify with the response’s own usage numbers

Every response carries a usage object with three fields that tell you exactly what happened, no server log required this time:

• usage.input_tokens: tokens processed at full price, not cached
• usage.cache_creation_input_tokens: tokens written to the cache this request, billed at the write premium
• usage.cache_read_input_tokens: tokens served from the cache this request, billed at roughly a tenth of normal price

print(response.usage.cache_creation_input_tokens)
print(response.usage.cache_read_input_tokens)
print(response.usage.input_tokens)

If cache_read_input_tokens reads zero across several requests that you believe share an identical prefix, you have the hosted-API equivalent of what Step 4 demonstrated locally: something in your prefix is not byte-identical between requests. The most common causes, in order of how often they show up in real code:

• A timestamp or a datetime.now() call embedded in the system prompt
• A per-request UUID or session ID placed before the cached content, exactly as in Step 4
• Serializing a dictionary to JSON without sorting keys, so two logically identical objects render as different byte strings
• A tool list that is rebuilt in a different order on every request

Why a lookback window matters in long agent conversations

Each cache breakpoint only looks back up to 20 content blocks to find a match. In a short chatbot exchange this never matters, but in a long agentic loop with many tool calls and tool results, a single turn can easily exceed 20 blocks. If caching works for the first several turns of an agent loop and then silently stops, this is very likely why. Place an additional breakpoint every 10 to 15 blocks in long-running turns rather than relying on a single marker at the very end.

Common Mistakes to Watch For

• Testing cache behavior with concurrent, parallel requests that share an identical prefix. The first response has to begin streaming before its cache entry becomes readable by anyone else, so simultaneous requests cannot read each other’s cache. Send one request, wait for the first token, then fire the rest.
• Assuming a fresh conversation will benefit from a cache written by a different, unrelated conversation just because the visible prompt text looks the same. As Step 4’s gotcha showed, this depends entirely on how your caching layer, whether a local single-slot Ollama server or a hosted provider’s cache store, actually scopes and keys its entries.
• Changing the tool list or switching models between requests. Both invalidate the cache from that point forward, because tools render before the system prompt in the request.
• Adding a cache marker to a prefix shorter than the model’s minimum cacheable length. It will not error; it will just quietly never cache.

Next Steps

You now have a repeatable, no-cost way to check whether any LLM integration you build is actually benefiting from prompt caching, along with the specific fix (reorder stable content before volatile content) for the single most common way it breaks. From here:

• If you run a production system with multiple backend replicas behind a load balancer, look into cache-aware routing, which directs a request to whichever backend replica most recently served that same prefix, rather than round-robining it to a cold one.
• If your application sees the same underlying question phrased many different ways, rather than the exact same prompt repeated, prompt caching cannot help; that is what a semantic cache, a separate lookup layer built on embeddings, is for.
• Add the usage.cache_read_input_tokens check from Step 6 to your own application’s logging or monitoring. It is a small addition that will tell you immediately if a future code change silently breaks caching you already rely on.