How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
Ollama 0.32.9 answers an oversized prompt with HTTP 200 and keeps only half a window, so this lab measures where the cut falls, makes Ollama refuse instead, sets num_ctx three ways, prices the...
You paste a long document into a local model, ask a question whose answer sits on the first page, and the model replies, politely and confidently, that the document does not say. Nothing crashed. The HTTP status was 200. The model is not the problem: the start of your prompt never reached it. Ollama, the program that runs the model, compared your prompt with a fixed-size context window (the amount of text the model can look at in one go, counted in tokens), found it too long, cut it down to fit, and carried on without a warning.
Table Of Content
- Before you start
- What you need
- A few words we will use
- The files at a glance
- Step 1: Find out which context window you really have
- Three documents, three numbers
- Write the client and ask the server
- Step 2: Reproduce the silent cut
- Build a prompt with one planted fact
- Ask the same question with two window sizes
- Step 3: Map what survives
- Step 4: The same cut through other doors
- Step 5: Make Ollama refuse instead of truncate
- Step 6: Set the window three ways, and prove each one
- Way 1: in the request
- Way 2: in a Modelfile
- Way 3: for the whole server
- Step 7: Price the window before you raise it
- Step 8: Wrap it in a guard you can test
- Test it without a model
- Check that the tests would catch a bug
- Run it against the real server
- Confirm everything works end to end
- Common mistakes and how to spot them
- Setting the variable where the server cannot see it
- Trusting a bigger number than the model supports
- Reading prompt_eval_count without knowing the real size
- Letting two clients fight over one model
- Raising the window until the model spills onto the CPU
- Where to go next
- Sources
This tutorial teaches you to find that window, to see exactly which part of a prompt gets cut, to make Ollama refuse instead of truncate, to set the window to the size you need in three different ways, to understand what a bigger window costs in memory, and to wrap all of it in a small Python function with tests. Every step uses a lab you build as you read, and the output printed under each script is the output I got. The model is a 1.5 billion parameter model, so each request finishes in seconds on a modest GPU.
One note on scope before we start. Every number here was measured on Ollama 0.32.9 running on Windows 11 with an NVIDIA RTX A400 that has 4 GiB of video memory. Ollama’s defaults depend on video memory and have changed between versions, so read the numbers as what this version did on this machine, and use the scripts to measure your own setup.
Before you start
What you need
You need Ollama 0.32.9 or a nearby version (check with ollama --version), the small model used throughout (ollama pull qwen2.5:1.5b), and Python 3.13 or newer. The lab uses only the Python standard library, plus pytest for the tests in Step 8. Any operating system works; the commands for setting environment variables differ between Windows and Linux or macOS, and both forms are shown where it matters. The Ollama server must be running. If the Ollama desktop app is open, it already is. Otherwise start one in a terminal with ollama serve and leave it running. You should be comfortable reading short Python scripts. No machine learning knowledge is assumed.
Put every file from this tutorial in one empty folder. The first line of each Python block is the file name, so you can copy the blocks straight into files.
A few words we will use
A token is the unit a model reads: a word, a piece of a word, or a single digit. In this lab one numbered log line of about 55 characters costs roughly 28 tokens, because numbers are often read one digit at a time. The context window is the maximum number of tokens the model can hold in memory at once, your prompt and its answer together. Ollama’s documentation puts it this way: “Context length is the maximum number of tokens that the model has access to in memory.” The setting that controls it is called num_ctx.
Every response from Ollama’s chat endpoint carries two counters. prompt_eval_count is the number of prompt tokens the model actually processed, and eval_count is the number of tokens it generated. The first one turns out to be the most useful clue in this whole tutorial. Finally, the KV cache is the memory the model uses to remember every token already in the window. It grows with the window size, which is why a bigger window costs memory (Step 7 measures it).
The files at a glance
| File | What it does | Used in |
|---|---|---|
olc.py |
A tiny Ollama client, plus a helper that reads Ollama’s “prompt too big” error | every step |
haystack.py |
Builds a long, boring prompt with one planted fact in it | Steps 2 to 8 |
step01_loaded_context.py to step07_cost.py |
One experiment per step, each printing a short table | Steps 1 to 7 |
ctxguard.py |
A function that sizes the window for you and refuses instead of truncating | Step 8 |
test_ctxguard.py, mutate.py |
Tests against a fake server, and a check that the tests catch planted bugs | Step 8 |
step08_guard_live.py |
The guard against the real server | Step 8 |
Step 1: Find out which context window you really have
Three documents, three numbers
Before measuring anything, it helps to know that Ollama’s own documentation gives three different answers to “what is the default window?”. The context length page says Ollama picks a default from your video memory: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. The FAQ says: “By default, Ollama uses a context window size of 4096 tokens.” The Modelfile reference describes num_ctx as “Sets the size of the context window used to generate the next token. (Default: 2048)”. When the server on this machine started, it printed a line that settles it for 0.32.9: vram-based default context total_vram="4.0 GiB" default_num_ctx=4096. The video-memory rule is the current one, and the 2048 in the Modelfile page looks like a leftover from an older version.
The lesson is not that the documentation is wrong, but that the default is a property of your machine, so you should read it from your own server instead of from a web page. That takes a short script, plus a tiny client we will reuse everywhere.
Write the client and ask the server
This is the whole client. call sends JSON and always returns a pair of the HTTP status and the parsed body, even for errors, so later scripts can look at failures without try blocks. chat wraps Ollama’s /api/chat endpoint with temperature 0 and a fixed seed so runs repeat. loaded reads /api/ps, which lists the models that are in memory right now. too_big is for Step 5, where you will see why it exists.
# olc.py
"""A tiny Ollama client for the lab. Standard library only."""
import json
import urllib.error
import urllib.request
HOST = "http://127.0.0.1:11434"
def call(path, body=None, host=HOST, timeout=900):
"""Return (http_status, parsed_json). Errors come back as (status, {"error": ...})."""
data = None if body is None else json.dumps(body).encode()
request = urllib.request.Request(host + path, data=data, headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
return response.status, json.load(response)
except urllib.error.HTTPError as err:
try:
return err.code, json.load(err)
except ValueError:
return err.code, {"error": str(err.reason)}
def chat(model, messages, host=HOST, options=None, **top):
body = {"model": model, "messages": messages, "stream": False, **top}
body["options"] = {"temperature": 0, "seed": 1, **(options or {})}
return call("/api/chat", body, host)
def loaded(host=HOST):
"""What /api/ps says is loaded right now."""
return call("/api/ps", host=host)[1].get("models", [])
def unload(model, host=HOST):
call("/api/generate", {"model": model, "keep_alive": 0}, host)
def too_big(status, data):
"""If Ollama refused a prompt for not fitting, return the details; otherwise None.
Ollama hands the runner's JSON error over as a string inside its own error field,
so it has to be parsed a second time.
"""
if status != 400:
return None
error = data.get("error")
try:
inner = json.loads(error)["error"] if isinstance(error, str) else error
except (ValueError, KeyError, TypeError):
return None
if isinstance(inner, dict) and inner.get("type") == "exceed_context_size_error":
return inner
return None
Now the first script. It unloads the model so we start clean, sends a trivial request so Ollama loads it with whatever default it picks, and prints what /api/ps reports. It also asks /api/show for the window the model was trained for, because a model cannot use more than that.
# step01_loaded_context.py
"""Which context window did Ollama really give the model?"""
from olc import call, chat, loaded, unload
MODEL = "qwen2.5:1.5b"
print("Ollama version:", call("/api/version")[1]["version"])
unload(MODEL)
chat(MODEL, [{"role": "user", "content": "Say hello."}], options={"num_predict": 8})
for m in loaded():
print(f"{m['name']}: context_length={m['context_length']}, size={m['size'] / 2**20:.0f} MiB, in VRAM={m['size_vram'] / 2**20:.0f} MiB")
shown = call("/api/show", {"model": MODEL})[1]
print("context length the model was trained for:", shown["model_info"]["qwen2.context_length"])
print("parameters baked into the model:", shown.get("parameters", "").split())
Ollama version: 0.32.9
qwen2.5:1.5b: context_length=4096, size=1112 MiB, in VRAM=1112 MiB
context length the model was trained for: 32768
parameters baked into the model: []
context_length=4096 is the number that matters. The model was trained for 32,768 tokens but Ollama loaded it with 4,096, one eighth of what it could use, because that is the default for a 4 GiB card. The command line shows the same thing. After running the script, call ollama ps and read the CONTEXT column:
NAME ID SIZE PROCESSOR CONTEXT UNTIL
qwen2.5:1.5b 65ec06548149 1.2 GB 100% GPU 4096 4 minutes from now
How to check this step worked: you can name the number in your CONTEXT column, and you know whether it came from a default or from a setting you made. If it is 4096 or 2048, a prompt of a few thousand words can already overflow it, as the next step shows.
Step 2: Reproduce the silent cut
Build a prompt with one planted fact
To see what a model can read, plant a fact where you can control its position and ask about it. The helper below builds a numbered log, replaces one line with the fact “The secret passphrase is MANGO-4417.”, and ends with a question. The log lines are numbered and different from each other on purpose: repeated lines would let a model guess its way through, and numbers make every line unique. The where argument is a position from 0.0 (the very first line) to 1.0 (the very last).
# haystack.py
"""Build a long, boring prompt with one planted fact in it."""
import random
QUESTION = "Question: what is the secret passphrase? Answer with the passphrase only."
def haystack(n_lines, needle=None, needle_at=None, seed=7):
"""n_lines numbered log lines; the needle replaces line number needle_at (0-based)."""
rng = random.Random(seed)
lines = []
for i in range(1, n_lines + 1):
lines.append(
f"Log line {i:05d}: sensor {rng.randint(10, 99)} read "
f"{rng.uniform(15, 25):.1f} degrees at {rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}."
)
if needle is not None:
lines[needle_at] = needle
return "\n".join(lines)
def prompt_with_needle(n_lines, where, code="MANGO-4417"):
"""where is a fraction from 0.0 (very start) to 1.0 (very end)."""
needle = f"The secret passphrase is {code}."
at = min(n_lines - 1, int(where * (n_lines - 1)))
return haystack(n_lines, needle, at) + "\n\n" + QUESTION
Ask the same question with two window sizes
The next script builds a 300-line prompt with the fact in the first line and asks for the passphrase twice: once with the default window and once with num_ctx set to 16,384 in the request’s options. Then it repeats the default-window question for prompts of different lengths. To print the true size of each prompt it uses a trick that Step 5 explains: it asks Ollama to refuse the prompt and reads the token count out of the refusal.
# step02_reproduce.py
"""Plant a fact at the start of a long prompt and see whether the model can still read it."""
from olc import chat, too_big
from haystack import prompt_with_needle
MODEL = "qwen2.5:1.5b"
def ask(prompt, **kw):
return chat(MODEL, [{"role": "user", "content": prompt}], **kw)
def real_tokens(prompt):
"""The true prompt size: ask Ollama to refuse (truncate=false) and read the number it reports."""
status, data = ask(prompt, truncate=False)
if status == 200:
return data["prompt_eval_count"]
return too_big(status, data)["n_prompt_tokens"]
prompt = prompt_with_needle(300, 0.0)
print(f"prompt: {len(prompt):,} characters, fact planted in the first line")
for label, options in [("default context", {}), ("num_ctx 16384", {"num_ctx": 16384})]:
status, data = ask(prompt, options=options)
print(f"{label:<16} HTTP {status} prompt_eval_count={data['prompt_eval_count']:<5} done_reason={data['done_reason']} answer={data['message']['content'].strip()!r}")
print()
print("lines real tokens prompt_eval_count found the fact?")
for lines in [100, 140, 145, 150, 160, 200, 300, 450, 600]:
prompt = prompt_with_needle(lines, 0.0)
status, data = ask(prompt)
found = "MANGO-4417" in data["message"]["content"]
print(f"{lines:<6} {real_tokens(prompt):<12,} {data['prompt_eval_count']:<18,} {'yes' if found else 'NO'}")
print()
print("window prompt_eval_count for the same 8,427-token prompt")
prompt = prompt_with_needle(300, 0.0)
for ctx in [2048, 4096, 8192]:
status, data = ask(prompt, options={"num_ctx": ctx})
print(f"{ctx:<8} {data['prompt_eval_count']:,}")
prompt: 16,257 characters, fact planted in the first line
default context HTTP 200 prompt_eval_count=2050 done_reason=stop answer='None provided in the log lines.'
num_ctx 16384 HTTP 200 prompt_eval_count=8427 done_reason=stop answer='MANGO-4417'
That is the failure in two lines. The default window answered “None provided in the log lines.” with HTTP 200 and done_reason=stop, as if everything were fine, while the larger window found MANGO-4417. Look at prompt_eval_count: the prompt is 8,427 tokens, and with the default window the model processed 2,050. Now the table that shows where the cliff is:
lines real tokens prompt_eval_count found the fact?
100 2,827 2,827 yes
140 3,947 3,947 yes
145 4,087 4,087 yes
150 4,227 2,050 NO
160 4,507 2,050 NO
200 5,627 2,050 NO
300 8,427 2,050 NO
450 12,627 2,050 NO
600 16,827 2,050 NO
Up to 4,087 tokens the prompt passes through untouched and the count is exact. At 4,227 tokens, which is only 131 over the window, the count does not become 4,096. It drops to 2,050, and it stays at 2,050 for every longer prompt, from 4,507 tokens up to 16,827. So the damage is not proportional: a prompt that is a little too long loses about half of itself. The last small table explains the number. With three different windows the same oversized prompt produced:
window prompt_eval_count for the same 8,427-token prompt
2048 1,026
4096 2,050
8192 4,098
In these runs the processed count was always half the window plus two. I did not read the code that does this, so treat “half the window plus two” as a measured rule for this model and this version, not a documented one. The practical meaning is the part to remember: when you see prompt_eval_count far below your real prompt size, text was dropped.
Common mistake: blaming the model. A 1.5 billion parameter model is weak, but in this test the 16,384 window shows it can read the fact perfectly well. The same model with the same question fails only when the text never arrives.
Step 3: Map what survives
Knowing that half a window survives, the next question is which half. The script below moves the planted fact through the same 8,427-token prompt in ten percent steps and asks for it each time, with the default window and with a 16,384 window as the control.
# step03_survival_map.py
"""Move the planted fact through an 8,427-token prompt and see which positions survive."""
from olc import chat
from haystack import prompt_with_needle
MODEL = "qwen2.5:1.5b"
print("position in prompt default (4096) num_ctx 16384")
for position in [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]:
prompt = prompt_with_needle(300, position)
cells = []
for options in ({}, {"num_ctx": 16384}):
status, data = chat(MODEL, [{"role": "user", "content": prompt}], options=options)
cells.append("found" if "MANGO-4417" in data["message"]["content"] else "LOST")
print(f"{position:>6.0%} {cells[0]:<15} {cells[1]}")
position in prompt default (4096) num_ctx 16384
0% LOST found
10% LOST found
20% LOST found
30% LOST found
40% LOST found
50% LOST found
60% LOST found
70% LOST found
80% found found
90% found found
100% found found
The control column says found in every row, so the model can read the fact anywhere when it can see it. The default column says LOST for every position up to 70 percent and found from 80 percent on. The cut keeps the end of the prompt and discards the beginning. The sizes agree: 20 percent of 8,427 tokens is about 1,700 tokens, which fits inside the 2,050 that survive, while 30 percent is about 2,500, which does not.
This has a practical consequence that is easy to miss. Instructions usually sit at the top of a prompt, because that is where people write them. Documents and chat history pile up below. A cut that keeps the bottom and drops the top removes exactly the part you wrote with the most care. Putting the question last, as this lab does, saves the question but not your instructions. This is one model and one prompt shape, so run the script with your own typical prompt before relying on the exact boundary.
Step 4: The same cut through other doors
Ollama has more than one place where text can be cut. The cut in Step 2 hit a single huge message, so it was made further down, after the prompt had been turned into tokens. For chats, the server adds a separate layer on top, and its source code explains the rule in a comment: “chatPrompt truncates any messages that exceed the context window of the model, making sure to always include 1) the latest message and 2) system messages”. In words: when a conversation is too long, the server drops the oldest messages first, keeps the system message and the newest message, and sends the rest on. If even that is too big, the lower layer cuts as in Step 2. The next script tests four ways in.
# step04_other_doors.py
"""The same silent cut through four other doors: chat history, a system message, the OpenAI-style endpoint, /api/generate."""
from olc import call, chat
from haystack import haystack, prompt_with_needle
MODEL = "qwen2.5:1.5b"
def show(label, status, data):
if status != 200:
print(f"{label:<28} HTTP {status}")
return
print(f"{label:<28} prompt_eval_count={data['prompt_eval_count']:<5} answer={data['message']['content'].strip()[:44]!r}")
print("1. A 40-turn chat; the codename is told in turn 1 and asked about in turn 41")
history = []
for turn in range(40):
text = haystack(6, seed=100 + turn)
if turn == 0:
text = "My project codename is BLUEHERON. Remember it." + chr(10) + text
history += [{"role": "user", "content": text}, {"role": "assistant", "content": "Noted."}]
history.append({"role": "user", "content": "What is my project codename? Answer with the codename only."})
show("default context", *chat(MODEL, history))
show("num_ctx 16384", *chat(MODEL, history, options={"num_ctx": 16384}))
print()
print("2. A system message with a fact, then one giant user message")
system = {"role": "system", "content": "Company policy: the refund window is 45 days."}
giant = haystack(300) + chr(10) * 2 + "Question: what is the refund window in days? Answer with the number only."
show("default context", *chat(MODEL, [system, {"role": "user", "content": giant}]))
show("num_ctx 16384", *chat(MODEL, [system, {"role": "user", "content": giant}], options={"num_ctx": 16384}))
print()
print("3. The OpenAI-compatible endpoint and 4. /api/generate, fact in the first line")
prompt = prompt_with_needle(300, 0.0)
status, data = call("/v1/chat/completions", {"model": MODEL, "messages": [{"role": "user", "content": prompt}], "temperature": 0, "seed": 1})
print(f"{'/v1/chat/completions':<28} usage.prompt_tokens={data['usage']['prompt_tokens']:<5} answer={data['choices'][0]['message']['content'][:44]!r}")
status, data = call("/api/generate", {"model": MODEL, "prompt": prompt, "stream": False, "options": {"temperature": 0, "seed": 1}})
print(f"{'/api/generate':<28} prompt_eval_count={data['prompt_eval_count']:<5} answer={data['response'].strip()[:44]!r}")
1. A 40-turn chat; the codename is told in turn 1 and asked about in turn 41
default context prompt_eval_count=4033 answer='Project Phoenix'
num_ctx 16384 prompt_eval_count=7295 answer='BLUEHERON'
2. A system message with a fact, then one giant user message
default context prompt_eval_count=2050 answer='5'
num_ctx 16384 prompt_eval_count=8441 answer='45'
3. The OpenAI-compatible endpoint and 4. /api/generate, fact in the first line
/v1/chat/completions usage.prompt_tokens=2050 answer='None provided in the log lines.'
/api/generate prompt_eval_count=2050 answer='None provided in the log lines.'
Read the four results one at a time. In the 40-turn chat, the codename told in the first turn was dropped as an old message. The prompt shrank to 4,033 tokens, just under the window, and the model, which had nothing to go on, made up “Project Phoenix” instead of saying it did not know. With a larger window the same conversation produced BLUEHERON. In the second test, a fact that lives in the system message, the company’s 45-day refund window, was lost because one giant user message pushed the whole prompt over the limit, and the cut removed the front of the prompt, which is where the system message sits. The model answered “5”. This is the case that matters for safety: a rule or a fact you place in the system message is not protected from a large enough user message. If a user, or a pasted document, can make the user message bigger than the window, the rules you wrote may never reach the model. The 45-day fact is a harmless stand-in, so replace it with a canary instruction of your own and test it the same way.
The last two lines show that the OpenAI-compatible endpoint and /api/generate behave the same way: the OpenAI-style response reports usage.prompt_tokens=2050 and the wrong answer. That endpoint has no options field, so you cannot raise the window per request there. Ollama’s documentation says so directly: “The OpenAI API does not have a way of setting the context size for a model.” For that endpoint you need one of the server-wide fixes from Step 6.
Step 5: Make Ollama refuse instead of truncate
Raising the window fixes the example prompts, but a bigger window is still finite, and real inputs keep growing. What you want is for an oversized prompt to fail loudly so your code can decide what to do. Ollama 0.32.9 has two request fields for this. In the chat and generate requests they are documented in the API types of its source code rather than on the API reference page I read. truncate: “Truncate is a boolean that, when set to true, truncates the chat history messages if the rendered prompt exceeds the context length limit.” shift: “Shift is a boolean that, when set to true, shifts the chat history when hitting the context length limit instead of erroring.” Both default to on for this model. Set them to false and see what happens.
# step05_fail_loud.py
"""Turn the silent cut into an error you can catch."""
from olc import chat, too_big
from haystack import prompt_with_needle
MODEL = "qwen2.5:1.5b"
messages = [{"role": "user", "content": prompt_with_needle(300, 0.0)}]
for label, flags in [("truncate=false", {"truncate": False}), ("shift=false", {"shift": False}), ("both false", {"truncate": False, "shift": False})]:
status, data = chat(MODEL, messages, **flags)
print(f"{label}: HTTP {status}")
print(" ", data["error"])
print()
status, data = chat(MODEL, messages, truncate=False)
details = too_big(status, data)
print("parsed from the error string:", {key: details[key] for key in ("type", "n_prompt_tokens", "n_ctx")})
status, data = chat(MODEL, messages, options={"num_ctx": 16384}, truncate=False, shift=False)
print(f"same flags with num_ctx 16384: HTTP {status}, prompt_eval_count={data['prompt_eval_count']}, answer={data['message']['content'].strip()!r}")
print()
print("When the window fills during the answer (a prompt of about 3,960 tokens, num_ctx 4096, num_predict 600):")
long_ask = prompt_with_needle(140, 0.5).replace("Answer with the passphrase only.", "Then print the numbers 1 to 600, one per line, never abbreviate.")
for label, flags in [("default", {}), ("truncate=false, shift=false", {"truncate": False, "shift": False})]:
status, data = chat(MODEL, [{"role": "user", "content": long_ask}], options={"num_ctx": 4096, "num_predict": 600}, **flags)
print(f"{label:<28} HTTP {status} prompt_eval_count={data['prompt_eval_count']} eval_count={data['eval_count']} done_reason={data['done_reason']}")
truncate=false: HTTP 400
{"error":{"code":400,"message":"request (8427 tokens) exceeds the available context size (4096 tokens), try increasing it","type":"exceed_context_size_error","n_prompt_tokens":8427,"n_ctx":4096}}
shift=false: HTTP 400
the prompt is longer than the context length currently available to the model; shorten the prompt, adjust the context length in settings, or use a model with a longer context length
both false: HTTP 400
{"error":{"code":400,"message":"request (8427 tokens) exceeds the available context size (4096 tokens), try increasing it","type":"exceed_context_size_error","n_prompt_tokens":8427,"n_ctx":4096}}
Each flag alone, and both together, turned the silent cut into an HTTP 400 error, which is what we wanted. Notice the odd shape of two of the answers: Ollama passes the underlying JSON error through as a string inside its own error field. That is why too_big in olc.py parses it a second time. The flag shift=false alone produces a plain sentence instead, with no numbers, so the guard in Step 8 sets both flags and reads the structured form.
The refusal carries the exact token count and window size, and it comes back without the model reading your text (Step 8 times it), so once the model is loaded it works as a nearly free token counter:
parsed from the error string: {'type': 'exceed_context_size_error', 'n_prompt_tokens': 8427, 'n_ctx': 4096}
same flags with num_ctx 16384: HTTP 200, prompt_eval_count=8427, answer='MANGO-4417'
The second line shows the same flags are harmless when the window is big enough: the request succeeds and the model finds the fact. The last part of the output covers a case that surprises people. The window holds your prompt and the model’s answer together, so even a prompt that fits can leave too little room to answer:
When the window fills during the answer (a prompt of about 3,960 tokens, num_ctx 4096, num_predict 600):
default HTTP 200 prompt_eval_count=3961 eval_count=600 done_reason=length
truncate=false, shift=false HTTP 200 prompt_eval_count=3961 eval_count=135 done_reason=length
With the defaults, the answer ran to the 600 tokens we allowed, because Ollama slides the window forward while it generates. With truncate=false, shift=false the answer stopped after 135 tokens, and 3,961 plus 135 is exactly 4,096. The reply is still HTTP 200 and done_reason=length in both rows, so the only clue is that eval_count (135) is smaller than the 600 tokens you allowed. The guard in Step 8 treats that pattern as “the window ran out” and retries with a bigger one.
Step 6: Set the window three ways, and prove each one
You can set num_ctx in three places. They differ in how far they reach, so it is worth knowing all three. The script after the list of ways tries each one against a prompt of 4,227 tokens, which is just over the default window, and asks /api/ps what was loaded.
Way 1: in the request
Put num_ctx in the options of a single request, as Step 2 did. It affects only that call, which makes it the safest way to experiment. The catch is that the window is a property of the loaded model, so a request that asks for a different size makes Ollama reload the model. The output below measures it.
Way 2: in a Modelfile
A Modelfile is a short recipe for a named model. Add the line PARAMETER num_ctx 8192 and every request to that model name gets the window with no options needed. This is also the fix for clients that cannot send options, such as anything that talks to the OpenAI-compatible endpoint. The documentation for that endpoint says: “If you need to change the context size, create a Modelfile”. The script creates a model called qwen2.5-1.5b-8k from a Modelfile and removes it again at the end.
Way 3: for the whole server
The environment variable OLLAMA_CONTEXT_LENGTH changes the default for every model the server loads. The FAQ gives the shell form, OLLAMA_CONTEXT_LENGTH=8192 ollama serve. In Windows PowerShell the same thing is $env:OLLAMA_CONTEXT_LENGTH = "8192"; ollama serve. If the Ollama desktop app starts your server, set the variable as the FAQ describes for Windows (“Ollama inherits your user and system environment variables”, so quit Ollama, edit your user environment variables, and start it again), or use the context length slider in the app’s settings that the context length page describes. To keep your running server untouched, the script starts a second server on port 11435 with the variable set and stops it afterwards.
# step06_set_it.py
"""Three ways to set the context window, each proven with /api/ps."""
import os
import subprocess
import tempfile
import time
from olc import HOST, call, chat, loaded, unload
from haystack import prompt_with_needle
MODEL = "qwen2.5:1.5b"
CUSTOM = "qwen2.5-1.5b-8k"
SECOND = "http://127.0.0.1:11435"
messages = [{"role": "user", "content": prompt_with_needle(150, 0.0)}] # 4,227 tokens: just over the 4096 default
def report(label, host=HOST, model=MODEL, **kw):
status, data = chat(model, messages, host, **kw)
window = [m["context_length"] for m in loaded(host)]
found = "MANGO-4417" in data["message"]["content"]
load = data["load_duration"] / 1e6
print(f"{label:<30} loaded context={window} prompt_eval_count={data['prompt_eval_count']:<5} fact found={found} model load {load:.0f} ms")
unload(MODEL)
report("nothing set")
unload(MODEL)
report("1. options.num_ctx = 8192", options={"num_ctx": 8192})
report(" same request again", options={"num_ctx": 8192})
report(" num_ctx = 16384, no unload", options={"num_ctx": 16384})
unload(MODEL)
folder = tempfile.mkdtemp()
modelfile = os.path.join(folder, "Modelfile")
with open(modelfile, "w", newline="\n") as handle:
handle.write(f"FROM {MODEL}\nPARAMETER num_ctx 8192\n")
subprocess.run(["ollama", "create", CUSTOM, "-f", modelfile], check=True, capture_output=True)
unload(MODEL)
report("2. Modelfile num_ctx 8192", model=CUSTOM)
print(" ollama show --parameters:", subprocess.run(["ollama", "show", CUSTOM, "--parameters"], capture_output=True, text=True).stdout.split())
unload(CUSTOM)
subprocess.run(["ollama", "rm", CUSTOM], check=True, capture_output=True)
env = dict(os.environ, OLLAMA_HOST="127.0.0.1:11435", OLLAMA_CONTEXT_LENGTH="8192")
server = subprocess.Popen(["ollama", "serve"], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
for _ in range(40):
try:
call("/api/version", host=SECOND, timeout=2)
break
except OSError:
time.sleep(0.5)
report("3. OLLAMA_CONTEXT_LENGTH=8192", host=SECOND)
unload(MODEL, SECOND)
finally:
subprocess.run(["taskkill", "/PID", str(server.pid), "/T", "/F"], capture_output=True)
nothing set loaded context=[4096] prompt_eval_count=2050 fact found=False model load 1781 ms
1. options.num_ctx = 8192 loaded context=[8192] prompt_eval_count=4227 fact found=True model load 1767 ms
same request again loaded context=[8192] prompt_eval_count=4227 fact found=True model load 144 ms
num_ctx = 16384, no unload loaded context=[16384] prompt_eval_count=4227 fact found=True model load 2582 ms
2. Modelfile num_ctx 8192 loaded context=[8192] prompt_eval_count=4227 fact found=True model load 2299 ms
ollama show --parameters: ['num_ctx', '8192']
3. OLLAMA_CONTEXT_LENGTH=8192 loaded context=[8192] prompt_eval_count=4227 fact found=True model load 1843 ms
Read the output from the top. With nothing set, the window is 4,096, the prompt is cut to 2,050 tokens, and the fact is not found. Way 1 loads an 8,192 window, the model processes all 4,227 tokens, and the fact is found. The last column is the important teaching detail: loading the model took 1,767 ms the first time, 144 ms for the same request again, and 2,582 ms when the next request asked for 16,384 without unloading first. A new window means a reload. If two programs share one Ollama server and ask for different sizes, they will make the model reload each time they take turns, so agree on one window per model. Way 2 prints num_ctx 8192 from ollama show --parameters, and Way 3 gives the same 8,192 window with no option anywhere in the request.
How to check this step worked: ollama ps shows your chosen number in the CONTEXT column while the model is loaded. If you set the environment variable and it still says 4096, the server you are talking to was started somewhere that did not see the variable. Set it where the server starts and restart it.
Step 7: Price the window before you raise it
A bigger window is not free. The model keeps a key and a value for every token in the window, in every layer, and that memory is reserved up front. The FAQ warns about the same effect from the other side: “Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH”, so the number of parallel requests multiplies it. The context length page puts the rule in one sentence: “Setting a larger context length will increase the amount of memory required to run a model.”
You can predict the cache size with arithmetic. Each token needs two numbers (a key and a value) per layer for each of the model’s key-value heads, and each number takes two bytes at the default 16-bit precision. The script reads the layer count and head sizes from /api/show, predicts the growth, and compares it with the size Ollama reports after loading the model at each window. It uses a prompt of about 1,550 tokens so that every row fits, asks for a single output token, and finally repeats the largest window with the cache quantized to 8 bits on a second server.
# step07_cost.py
"""What a bigger context window costs in memory and time."""
import os
import subprocess
import time
from olc import HOST, call, chat, loaded, unload
from haystack import prompt_with_needle
MODEL = "qwen2.5:1.5b"
SECOND = "http://127.0.0.1:11435"
info = call("/api/show", {"model": MODEL})[1]["model_info"]
layers = info["qwen2.block_count"]
kv_heads = info["qwen2.attention.head_count_kv"]
head_dim = info["qwen2.embedding_length"] // info["qwen2.attention.head_count"]
per_token = 2 * layers * kv_heads * head_dim * 2 # K and V, 2 bytes per number in f16
print(f"layers={layers}, kv_heads={kv_heads}, head_dim={head_dim}: {per_token:,} bytes of KV cache per token")
messages = [{"role": "user", "content": prompt_with_needle(55, 0.0)}] # about 1,550 tokens, fits every row
def measure(ctx, host=HOST):
unload(MODEL, host)
status, data = chat(MODEL, messages, host, options={"num_ctx": ctx, "num_predict": 1})
entry = loaded(host)[0]
speed = data["prompt_eval_count"] / (data["prompt_eval_duration"] / 1e9)
return entry["context_length"], entry["size"] / 2**20, data["load_duration"] / 1e6, speed
print()
print("asked loaded loaded size growth vs 2048 KV arithmetic load time prefill")
first = None
for ctx in [2048, 4096, 8192, 16384, 32768, 65536]:
window, size, load_ms, speed = measure(ctx)
first = first or size
arithmetic = (window - 2048) * per_token / 2**20
print(f"{ctx:<9} {window:<11} {size:>7.0f} MiB {size - first:>8.0f} MiB {arithmetic:>8.0f} MiB {load_ms:>6.0f} ms {speed:>5.0f} tok/s")
unload(MODEL)
env = dict(os.environ, OLLAMA_HOST="127.0.0.1:11435", OLLAMA_FLASH_ATTENTION="1", OLLAMA_KV_CACHE_TYPE="q8_0")
server = subprocess.Popen(["ollama", "serve"], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
for _ in range(40):
try:
call("/api/version", host=SECOND, timeout=2)
break
except OSError:
time.sleep(0.5)
window, size, load_ms, speed = measure(32768, SECOND)
print()
print(f"32768 with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0: {size:.0f} MiB, load {load_ms:.0f} ms, prefill {speed:.0f} tok/s")
unload(MODEL, SECOND)
finally:
subprocess.run(["taskkill", "/PID", str(server.pid), "/T", "/F"], capture_output=True)
layers=28, kv_heads=2, head_dim=128: 28,672 bytes of KV cache per token
asked loaded loaded size growth vs 2048 KV arithmetic load time prefill
2048 2048 1054 MiB 0 MiB 0 MiB 1757 ms 2000 tok/s
4096 4096 1112 MiB 58 MiB 56 MiB 1762 ms 2000 tok/s
8192 8192 1298 MiB 244 MiB 168 MiB 1749 ms 2003 tok/s
16384 16384 1538 MiB 484 MiB 392 MiB 1777 ms 1976 tok/s
32768 32768 2018 MiB 964 MiB 840 MiB 1755 ms 2007 tok/s
65536 32768 2018 MiB 964 MiB 840 MiB 1739 ms 2002 tok/s
32768 with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0: 1598 MiB, load 1830 ms, prefill 1939 tok/s
This model has 28 layers, 2 key-value heads and a head size of 128, which makes 2 × 28 × 2 × 128 × 2 = 28,672 bytes of cache per token. From a 2,048 window to a 32,768 window the loaded size grew by 964 MiB. The arithmetic predicts 840 MiB for the cache alone, so the measured growth is higher in every row, by 2 to 124 MiB. I did not break down the difference; other buffers that also scale with the window are the likely reason. The size almost doubled, from 1,054 to 2,018 MiB, while the model file itself is under 1 GB. On this card the load time (about 1.75 seconds) and the prefill speed (about 2,000 tokens per second) did not change from row to row for this short prompt, so the price here is memory. I did not measure a prompt that fills a large window, where the work per token grows, so measure that with your real prompts.
Two more lines in the table deserve attention. Asking for 65,536 loaded 32,768, the model’s trained window, at exactly the same size as the row above, so a request for more than the model was trained for is lowered without any message. And the last line shows what the cache type buys you. The FAQ describes the setting: “The K/V context cache can be quantized to significantly reduce memory usage when Flash Attention is enabled.” With OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, the 32,768 window loaded at 1,598 MiB instead of 2,018 MiB, a saving of 420 MiB. The arithmetic agrees: a 32,768-token cache at 2 bytes per number is 896 MiB, the 8-bit format stores 32 numbers in 34 bytes (a 2-byte scale plus 32 one-byte values, as I understand ggml’s q8_0 layout), so it needs about 476 MiB, and 896 minus 476 leaves a saving of 420 MiB. The FAQ says q8_0 “uses approximately 1/2 the memory of f16 with a very small loss in precision”. That is a setting for the whole server, so test your model’s answers before adopting it.
That gives you a way to choose a window. Count the tokens in your longest realistic prompt (Step 5 gives the exact number), add the longest answer you will allow, add a margin, and round up to a standard size. Then load the model and read ollama ps. The context length page advises: “For best performance, use the maximum context length for a model, and avoid offloading the model to CPU. Verify the split under PROCESSOR using ollama ps.” If PROCESSOR shows part of the model on the CPU, your window is too big for your card, and lowering it is usually the better trade than accepting the slowdown. The same page says tasks such as agents and coding tools should use at least 64,000 tokens, which a model trained for 32,768 cannot offer, so that advice only applies to models with a longer trained window.
Step 8: Wrap it in a guard you can test
Steps 5 to 7 give you every piece of a safe call: refuse oversized prompts, read the exact size from the refusal, choose a window that fits, and notice when the window ran out during an answer. This step assembles them into one function so that no caller has to remember the rules. The design is deliberately small:
The guard starts at a cheap 4,096 window and always sends truncate=false and shift=false, so Ollama can only answer in full or refuse. If it refuses, the guard reads the prompt size from the refusal, adds the tokens you reserved for the answer, and picks the smallest standard window that holds both. If that window is above your budget (max_ctx), it raises PromptTooLong before any generation work is done. If a reply comes back with done_reason=length but fewer tokens than you allowed, the window filled during the answer, and the guard retries with a bigger one. Any other failure raises OllamaError. It tries at most three times.
# ctxguard.py
"""Call Ollama so that a prompt is either read in full or refused. Never cut silently."""
from olc import HOST, chat, too_big
TIERS = (2048, 4096, 8192, 16384, 32768, 65536, 131072)
class PromptTooLong(Exception):
def __init__(self, tokens, budget):
super().__init__(f"prompt is {tokens} tokens but the budget is {budget}")
self.tokens = tokens
self.budget = budget
class OllamaError(Exception):
pass
def pick_context(needed, max_ctx):
"""The smallest standard window that holds `needed` tokens, or None if it would pass max_ctx."""
for tier in TIERS:
if tier >= needed:
return tier if tier <= max_ctx else None
return None
def window_ran_out(status, data, answer_tokens):
"""A reply that stops early with done_reason 'length' ended because the window filled, not because of num_predict."""
return status == 200 and data.get("done_reason") == "length" and data.get("eval_count", 0) < answer_tokens
def safe_chat(model, messages, *, max_ctx=16384, answer_tokens=512, start_ctx=4096, host=HOST, options=None):
"""Return a complete reply, or raise. Sizes num_ctx from what Ollama reports; never lets it truncate or shift."""
base = {**(options or {}), "num_predict": answer_tokens}
ctx = start_ctx
for _ in range(3):
status, data = chat(model, messages, host, options={**base, "num_ctx": ctx}, truncate=False, shift=False)
refused = too_big(status, data)
if refused:
prompt_tokens = refused["n_prompt_tokens"]
elif window_ran_out(status, data, answer_tokens):
prompt_tokens = data["prompt_eval_count"]
elif status == 200:
return data
else:
raise OllamaError(f"HTTP {status}: {data.get('error')}")
ctx = pick_context(prompt_tokens + answer_tokens, max_ctx)
if ctx is None:
raise PromptTooLong(prompt_tokens, max_ctx - answer_tokens)
raise OllamaError("could not size the context window in three tries")
Test it without a model
You do not need a language model to test this logic, only something that answers like Ollama does. The test file below starts a tiny fake server in a thread. It charges 20 tokens per message plus one token per word, refuses oversized prompts the way the real server does (JSON inside a string), and silently cuts them to half a window plus two when the flags are missing. That last detail matters: it means a guard that forgets the flags gets a wrong answer from the fake server, and a test can notice. Each test checks one rule from the design above.
# test_ctxguard.py
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import pytest
import ctxguard
class FakeOllama:
"""Pretends to be /api/chat. A message of N words costs N + 20 tokens; the model wants to write answer_tokens tokens."""
def __init__(self):
self.calls = []
self.answer_tokens = 30
self.fail_with = None
outer = self
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
status, payload = outer.respond(body)
raw = json.dumps(payload).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
self.url = f"http://127.0.0.1:{self.server.server_address[1]}"
threading.Thread(target=self.server.serve_forever, daemon=True).start()
def respond(self, body):
options = body["options"]
ctx = options["num_ctx"]
self.calls.append({"num_ctx": ctx, "truncate": body.get("truncate"), "shift": body.get("shift"), "temperature": options.get("temperature")})
if self.fail_with:
return self.fail_with, {"error": "boom"}
tokens = sum(len(m["content"].split()) + 20 for m in body["messages"])
strict = body.get("truncate") is False and body.get("shift") is False
if tokens > ctx:
if strict:
refusal = {"error": {"code": 400, "message": "too big", "type": "exceed_context_size_error", "n_prompt_tokens": tokens, "n_ctx": ctx}}
return 400, {"error": json.dumps(refusal)} # like the real server: JSON inside a string
return 200, {"message": {"content": "ok"}, "prompt_eval_count": ctx // 2 + 2, "eval_count": 5, "done_reason": "stop"}
used = min(self.answer_tokens, options["num_predict"], ctx - tokens)
return 200, {"message": {"content": "ok"}, "prompt_eval_count": tokens, "eval_count": used, "done_reason": "stop" if used == self.answer_tokens else "length"}
@pytest.fixture
def fake():
return FakeOllama()
def words(count):
return [{"role": "user", "content": " ".join(["word"] * count)}]
def run(fake, count, **kw):
return ctxguard.safe_chat("m", words(count), host=fake.url, **kw)
def test_short_prompt_takes_one_call_at_the_start_window(fake):
data = run(fake, 100)
assert data["done_reason"] == "stop"
assert [c["num_ctx"] for c in fake.calls] == [4096]
def test_big_prompt_is_refused_then_sized_from_the_reported_count(fake):
data = run(fake, 6000) # 6,020 tokens plus 512 for the answer needs 8192
assert [c["num_ctx"] for c in fake.calls] == [4096, 8192]
assert data["prompt_eval_count"] == 6020
def test_prompt_over_budget_raises_after_one_cheap_call(fake):
with pytest.raises(ctxguard.PromptTooLong) as caught:
run(fake, 40000, max_ctx=16384)
assert caught.value.tokens == 40020
assert len(fake.calls) == 1
def test_every_call_forbids_truncating_and_shifting(fake):
run(fake, 6000)
assert all(c["truncate"] is False and c["shift"] is False for c in fake.calls)
def test_window_that_fills_during_the_answer_is_retried_bigger(fake):
fake.answer_tokens = 400
data = run(fake, 3800, answer_tokens=512) # 3,820 prompt tokens leave 276 of 4096
assert [c["num_ctx"] for c in fake.calls] == [4096, 8192]
assert data["done_reason"] == "stop" and data["eval_count"] == 400
def test_unexpected_server_error_is_raised_not_swallowed(fake):
fake.fail_with = 500
with pytest.raises(ctxguard.OllamaError):
run(fake, 10)
def test_other_400_errors_are_not_mistaken_for_size_refusals(fake):
fake.fail_with = 400
with pytest.raises(ctxguard.OllamaError):
run(fake, 10)
assert len(fake.calls) == 1
def test_callers_options_still_reach_the_server(fake):
run(fake, 10, options={"temperature": 0.3})
assert fake.calls[0]["temperature"] == 0.3
@pytest.mark.parametrize(
"needed, max_ctx, expected",
[(2048, 131072, 2048), (2049, 131072, 4096), (4096, 131072, 4096), (4097, 16384, 8192), (16385, 16384, None), (200000, 10**9, None)],
)
def test_pick_context_boundaries(needed, max_ctx, expected):
assert ctxguard.pick_context(needed, max_ctx) == expected
.............. [100%]
14 passed in 0.09s
Check that the tests would catch a bug
Passing tests prove little if they cannot fail. The script below plants eight bugs in ctxguard.py, one at a time, runs the tests against each, and restores the file. Every bug should make at least one test fail.
# mutate.py
"""Plant one bug at a time in ctxguard.py and check that the tests notice."""
import os
import pathlib
import shutil
import subprocess
import sys
SOURCE = pathlib.Path("ctxguard.py")
MUTANTS = [
("allow truncation", "truncate=False, shift=False)", "truncate=True, shift=False)"),
("allow shifting", "truncate=False, shift=False)", "truncate=False, shift=True)"),
("tier boundary off by one", "if tier >= needed:", "if tier > needed:"),
("ignore the answer budget", "pick_context(prompt_tokens + answer_tokens, max_ctx)", "pick_context(prompt_tokens, max_ctx)"),
("ignore max_ctx", "return tier if tier <= max_ctx else None", "return tier"),
("trust a reply that ran out of window", "elif window_ran_out(status, data, answer_tokens):", "elif False:"),
("never recognise a size refusal", "if refused:", "if False:"),
("swallow server errors", 'raise OllamaError(f"HTTP', 'return data # raise OllamaError(f"HTTP'),
]
def run_tests():
shutil.rmtree("__pycache__", ignore_errors=True)
env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", COLUMNS="170")
result = subprocess.run(
[sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "test_ctxguard.py"],
capture_output=True, text=True, env=env,
)
failed = [line for line in result.stdout.splitlines() if line.startswith("FAILED")]
return result.returncode, len(failed)
original = SOURCE.read_text()
code, _ = run_tests()
print(f"clean source: tests exit code {code}")
caught = 0
try:
for name, old, new in MUTANTS:
assert original.count(old) == 1, name
SOURCE.write_text(original.replace(old, new))
code, failed = run_tests()
caught += code != 0
print(f"{name:<40} {failed} failing tests -> {'caught' if code else 'SURVIVED'}")
finally:
SOURCE.write_text(original)
print(f"{caught} of {len(MUTANTS)} planted bugs caught")
clean source: tests exit code 0
allow truncation 3 failing tests -> caught
allow shifting 3 failing tests -> caught
tier boundary off by one 2 failing tests -> caught
ignore the answer budget 1 failing tests -> caught
ignore max_ctx 2 failing tests -> caught
trust a reply that ran out of window 1 failing tests -> caught
never recognise a size refusal 3 failing tests -> caught
swallow server errors 2 failing tests -> caught
8 of 8 planted bugs caught
All eight were caught. The two flag mutants fail three tests each, because the fake server then truncates silently and the guard’s answers go wrong. The mutants in the sizing code (the tier boundary, the answer reserve and the budget) fail only one or two tests each, which shows that each of those rules is pinned down by a test written for it. This is also how I know the window-filled test earns its place: without it, the “trust a reply that ran out of window” bug would survive.
Run it against the real server
Finally, three calls to the real model: a short question, the 8,427-token prompt from Step 2, and a prompt of about 25,000 tokens that is over the budget.
# step08_guard_live.py
"""Run the guard against the real server."""
import time
from ctxguard import PromptTooLong, safe_chat
from haystack import prompt_with_needle
from olc import loaded, unload
MODEL = "qwen2.5:1.5b"
unload(MODEL)
def ask(label, text):
started = time.time()
try:
data = safe_chat(MODEL, [{"role": "user", "content": text}])
except PromptTooLong as err:
print(f"{label:<22} refused: {err} ({time.time() - started:.1f} s)")
return
window = [m["context_length"] for m in loaded()]
print(f"{label:<22} window={window} prompt_eval_count={data['prompt_eval_count']:<6} answer={data['message']['content'].strip()[:30]!r} ({time.time() - started:.1f} s)")
ask("short question", "Name the capital of France in one word.")
ask("8,427-token prompt", prompt_with_needle(300, 0.0))
ask("25,000-token prompt", prompt_with_needle(900, 0.0))
ask("25,000 tokens, asked again", prompt_with_needle(900, 0.0))
short question window=[4096] prompt_eval_count=38 answer='Paris.' (1.9 s)
8,427-token prompt window=[16384] prompt_eval_count=8427 answer='MANGO-4417' (7.5 s)
25,000-token prompt refused: prompt is 25227 tokens but the budget is 15872 (2.6 s)
25,000 tokens, asked again refused: prompt is 25227 tokens but the budget is 15872 (0.2 s)
The short question stayed at the 4,096 window and cost one call. The 8,427-token prompt was refused at 4,096, then answered correctly after the guard moved to 16,384 (8,427 plus 512 reserved tokens needs more than 8,192). The 25,000-token prompt was refused at once with a clear message, “prompt is 25227 tokens but the budget is 15872”, which is the 16,384 budget minus the 512 tokens reserved for the answer. That first refusal took 2.6 seconds, because the model had to reload at the small starting window. The fourth line asks the same question again with the model already loaded that way, and the refusal came back in 0.2 seconds, which is the real cost of a rejection: it arrives before the model reads your text.
Confirm everything works end to end
Run the full chain once more as a final check. In the lab folder, run python step02_reproduce.py and confirm you can reproduce the cliff on your own machine, with prompt_eval_count collapsing for prompts that are larger than your window. Run python -m pytest -q test_ctxguard.py and confirm all tests pass. Run python step08_guard_live.py and confirm the short question stays small, the large prompt gets a larger window and the right answer, and the oversized prompt is refused. Finally, call ollama ps and read the CONTEXT column once more. If those four checks agree with what this page printed, your setup can no longer cut a prompt silently.
Common mistakes and how to spot them
Setting the variable where the server cannot see it
You export OLLAMA_CONTEXT_LENGTH in one terminal, but the server was started by the desktop app or a service, so it never saw the variable. The tell is ollama ps still showing 4096. Set the variable where the server starts, restart the server, and check the CONTEXT column.
Trusting a bigger number than the model supports
Asking for 65,536 on a model trained for 32,768 gave a 32,768 window with no message, so a pipeline that assumes 65,536 will still be cut. Compare the window you asked for with the context_length in /api/ps after the first request.
Reading prompt_eval_count without knowing the real size
The count only exposes a cut if you know how many tokens you sent. Log it next to the exact size from the refusal trick, or use the guard so the comparison is not left to chance.
Letting two clients fight over one model
Different window sizes make the model reload, which Step 6 timed at about 2.6 seconds against 144 milliseconds for a repeat. Pick one window per model name and give each use case its own Modelfile name if they truly need different sizes.
Raising the window until the model spills onto the CPU
The context length page tells you to avoid offloading the model to the CPU and to check the PROCESSOR column. If you see a CPU share, lower the window or quantize the cache, as Step 7 showed, and test again.
Where to go next
Replace the log-line prompt with your own typical prompt and rerun Steps 2 and 3 to find where your cliff is. Add the guard to the code that calls Ollama and log every PromptTooLong, because each one is a prompt that used to be cut without anyone noticing. If you use Ollama through an OpenAI-compatible client, move the window into a Modelfile or the server setting and keep the guard’s native-API calls for checking. For related ground on this site, diagnosing a low prompt cache hit rate with Ollama covers what the server remembers between requests, benchmarking inference latency with Python and Ollama covers how to time what you change, and stopping PII leaks and prompt injection with guardrails covers rules that, as Step 4 showed, only help if they stay inside the window. Our agent harness tutorial ran into the same 2,050-token ceiling in a context probe.
Sources
The documentation quoted above, and the source files read for this lab, are all public:
- Ollama documentation: Context length, FAQ, Modelfile reference and OpenAI compatibility.
- Ollama source code at the v0.32.9 tag: server/prompt.go (message truncation), api/types.go (the
truncateandshiftfields) and server/routes.go (the video-memory default for the window).







No Comment! Be the first one.