TRENDING
A black manual telephone switchboard with rows of orange jacks and red patch cords plugged into the lower row, above a rotary dial and a handset
November 16, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
A brass nail with a threaded screw section lying on graph paper, a fastener that is both nail and screw
November 16, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
An amber warning beacon on a white surface, a flashing-light stand-in for the FBI’s FLASH alert
November 16, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query
Two eggs and a colour-changing egg timer in a saucepan of water on an induction hob
November 16, 2026
How to Set gRPC Deadlines and Retries in Python, Pass Them Down a Call Chain, and Stop Retries From Charging a Card Twice
A whole lemon and a halved lemon side by side on a black background
November 16, 2026
IBM’s LLMON Turns Prompt-Injection Defense Into a Token Layout, but the “Nearly Zero” Chart Is One Benchmark, the Paper’s Own Red Team Lets 15% to 20% of Indirect Attacks Through, and the Shipped Template Lets a Document Type the Delimiters
16 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
A manual paper cutter with a wooden grid board, a clear plastic guard and a long cutting arm, a picture of text being trimmed to fit
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
A brown llama and her gray cria walking across a salt flat beside a red lake in Bolivia
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
A stack of chocolate chip cookies on a white plate beside a cup of tea
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 16, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 302 Posts
News 307 Posts
Learning Hub 274 Posts
Home/Learning Hub/How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
Learning Hub

How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory

Ollama 0.32.9 answers an oversized prompt with HTTP 200 and keeps only half a window, so this lab measures where the cut falls, makes Ollama refuse instead, sets num_ctx three ways, prices the...

November 16, 2026 31 Min Read
8

You paste a long document into a local model, ask a question whose answer sits on the first page, and the model replies, politely and confidently, that the document does not say. Nothing crashed. The HTTP status was 200. The model is not the problem: the start of your prompt never reached it. Ollama, the program that runs the model, compared your prompt with a fixed-size context window (the amount of text the model can look at in one go, counted in tokens), found it too long, cut it down to fit, and carried on without a warning.

Table Of Content

  • Before you start
  • What you need
  • A few words we will use
  • The files at a glance
  • Step 1: Find out which context window you really have
  • Three documents, three numbers
  • Write the client and ask the server
  • Step 2: Reproduce the silent cut
  • Build a prompt with one planted fact
  • Ask the same question with two window sizes
  • Step 3: Map what survives
  • Step 4: The same cut through other doors
  • Step 5: Make Ollama refuse instead of truncate
  • Step 6: Set the window three ways, and prove each one
  • Way 1: in the request
  • Way 2: in a Modelfile
  • Way 3: for the whole server
  • Step 7: Price the window before you raise it
  • Step 8: Wrap it in a guard you can test
  • Test it without a model
  • Check that the tests would catch a bug
  • Run it against the real server
  • Confirm everything works end to end
  • Common mistakes and how to spot them
  • Setting the variable where the server cannot see it
  • Trusting a bigger number than the model supports
  • Reading prompt_eval_count without knowing the real size
  • Letting two clients fight over one model
  • Raising the window until the model spills onto the CPU
  • Where to go next
  • Sources

This tutorial teaches you to find that window, to see exactly which part of a prompt gets cut, to make Ollama refuse instead of truncate, to set the window to the size you need in three different ways, to understand what a bigger window costs in memory, and to wrap all of it in a small Python function with tests. Every step uses a lab you build as you read, and the output printed under each script is the output I got. The model is a 1.5 billion parameter model, so each request finishes in seconds on a modest GPU.

One note on scope before we start. Every number here was measured on Ollama 0.32.9 running on Windows 11 with an NVIDIA RTX A400 that has 4 GiB of video memory. Ollama’s defaults depend on video memory and have changed between versions, so read the numbers as what this version did on this machine, and use the scripts to measure your own setup.

Before you start

What you need

You need Ollama 0.32.9 or a nearby version (check with ollama --version), the small model used throughout (ollama pull qwen2.5:1.5b), and Python 3.13 or newer. The lab uses only the Python standard library, plus pytest for the tests in Step 8. Any operating system works; the commands for setting environment variables differ between Windows and Linux or macOS, and both forms are shown where it matters. The Ollama server must be running. If the Ollama desktop app is open, it already is. Otherwise start one in a terminal with ollama serve and leave it running. You should be comfortable reading short Python scripts. No machine learning knowledge is assumed.

Put every file from this tutorial in one empty folder. The first line of each Python block is the file name, so you can copy the blocks straight into files.

A few words we will use

A token is the unit a model reads: a word, a piece of a word, or a single digit. In this lab one numbered log line of about 55 characters costs roughly 28 tokens, because numbers are often read one digit at a time. The context window is the maximum number of tokens the model can hold in memory at once, your prompt and its answer together. Ollama’s documentation puts it this way: “Context length is the maximum number of tokens that the model has access to in memory.” The setting that controls it is called num_ctx.

Every response from Ollama’s chat endpoint carries two counters. prompt_eval_count is the number of prompt tokens the model actually processed, and eval_count is the number of tokens it generated. The first one turns out to be the most useful clue in this whole tutorial. Finally, the KV cache is the memory the model uses to remember every token already in the window. It grows with the window size, which is why a bigger window costs memory (Step 7 measures it).

The files at a glance

File What it does Used in
olc.py A tiny Ollama client, plus a helper that reads Ollama’s “prompt too big” error every step
haystack.py Builds a long, boring prompt with one planted fact in it Steps 2 to 8
step01_loaded_context.py to step07_cost.py One experiment per step, each printing a short table Steps 1 to 7
ctxguard.py A function that sizes the window for you and refuses instead of truncating Step 8
test_ctxguard.py, mutate.py Tests against a fake server, and a check that the tests catch planted bugs Step 8
step08_guard_live.py The guard against the real server Step 8

Step 1: Find out which context window you really have

Three documents, three numbers

Before measuring anything, it helps to know that Ollama’s own documentation gives three different answers to “what is the default window?”. The context length page says Ollama picks a default from your video memory: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. The FAQ says: “By default, Ollama uses a context window size of 4096 tokens.” The Modelfile reference describes num_ctx as “Sets the size of the context window used to generate the next token. (Default: 2048)”. When the server on this machine started, it printed a line that settles it for 0.32.9: vram-based default context total_vram="4.0 GiB" default_num_ctx=4096. The video-memory rule is the current one, and the 2048 in the Modelfile page looks like a leftover from an older version.

The lesson is not that the documentation is wrong, but that the default is a property of your machine, so you should read it from your own server instead of from a web page. That takes a short script, plus a tiny client we will reuse everywhere.

Write the client and ask the server

This is the whole client. call sends JSON and always returns a pair of the HTTP status and the parsed body, even for errors, so later scripts can look at failures without try blocks. chat wraps Ollama’s /api/chat endpoint with temperature 0 and a fixed seed so runs repeat. loaded reads /api/ps, which lists the models that are in memory right now. too_big is for Step 5, where you will see why it exists.

# olc.py
"""A tiny Ollama client for the lab. Standard library only."""
import json
import urllib.error
import urllib.request

HOST = "http://127.0.0.1:11434"


def call(path, body=None, host=HOST, timeout=900):
    """Return (http_status, parsed_json). Errors come back as (status, {"error": ...})."""
    data = None if body is None else json.dumps(body).encode()
    request = urllib.request.Request(host + path, data=data, headers={"Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return response.status, json.load(response)
    except urllib.error.HTTPError as err:
        try:
            return err.code, json.load(err)
        except ValueError:
            return err.code, {"error": str(err.reason)}


def chat(model, messages, host=HOST, options=None, **top):
    body = {"model": model, "messages": messages, "stream": False, **top}
    body["options"] = {"temperature": 0, "seed": 1, **(options or {})}
    return call("/api/chat", body, host)


def loaded(host=HOST):
    """What /api/ps says is loaded right now."""
    return call("/api/ps", host=host)[1].get("models", [])


def unload(model, host=HOST):
    call("/api/generate", {"model": model, "keep_alive": 0}, host)


def too_big(status, data):
    """If Ollama refused a prompt for not fitting, return the details; otherwise None.

    Ollama hands the runner's JSON error over as a string inside its own error field,
    so it has to be parsed a second time.
    """
    if status != 400:
        return None
    error = data.get("error")
    try:
        inner = json.loads(error)["error"] if isinstance(error, str) else error
    except (ValueError, KeyError, TypeError):
        return None
    if isinstance(inner, dict) and inner.get("type") == "exceed_context_size_error":
        return inner
    return None

Now the first script. It unloads the model so we start clean, sends a trivial request so Ollama loads it with whatever default it picks, and prints what /api/ps reports. It also asks /api/show for the window the model was trained for, because a model cannot use more than that.

# step01_loaded_context.py
"""Which context window did Ollama really give the model?"""
from olc import call, chat, loaded, unload

MODEL = "qwen2.5:1.5b"

print("Ollama version:", call("/api/version")[1]["version"])
unload(MODEL)
chat(MODEL, [{"role": "user", "content": "Say hello."}], options={"num_predict": 8})
for m in loaded():
    print(f"{m['name']}: context_length={m['context_length']}, size={m['size'] / 2**20:.0f} MiB, in VRAM={m['size_vram'] / 2**20:.0f} MiB")
shown = call("/api/show", {"model": MODEL})[1]
print("context length the model was trained for:", shown["model_info"]["qwen2.context_length"])
print("parameters baked into the model:", shown.get("parameters", "").split())
Ollama version: 0.32.9
qwen2.5:1.5b: context_length=4096, size=1112 MiB, in VRAM=1112 MiB
context length the model was trained for: 32768
parameters baked into the model: []

context_length=4096 is the number that matters. The model was trained for 32,768 tokens but Ollama loaded it with 4,096, one eighth of what it could use, because that is the default for a 4 GiB card. The command line shows the same thing. After running the script, call ollama ps and read the CONTEXT column:

NAME            ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
qwen2.5:1.5b    65ec06548149    1.2 GB    100% GPU     4096       4 minutes from now    

How to check this step worked: you can name the number in your CONTEXT column, and you know whether it came from a default or from a setting you made. If it is 4096 or 2048, a prompt of a few thousand words can already overflow it, as the next step shows.

Step 2: Reproduce the silent cut

Build a prompt with one planted fact

To see what a model can read, plant a fact where you can control its position and ask about it. The helper below builds a numbered log, replaces one line with the fact “The secret passphrase is MANGO-4417.”, and ends with a question. The log lines are numbered and different from each other on purpose: repeated lines would let a model guess its way through, and numbers make every line unique. The where argument is a position from 0.0 (the very first line) to 1.0 (the very last).

# haystack.py
"""Build a long, boring prompt with one planted fact in it."""
import random

QUESTION = "Question: what is the secret passphrase? Answer with the passphrase only."


def haystack(n_lines, needle=None, needle_at=None, seed=7):
    """n_lines numbered log lines; the needle replaces line number needle_at (0-based)."""
    rng = random.Random(seed)
    lines = []
    for i in range(1, n_lines + 1):
        lines.append(
            f"Log line {i:05d}: sensor {rng.randint(10, 99)} read "
            f"{rng.uniform(15, 25):.1f} degrees at {rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}."
        )
    if needle is not None:
        lines[needle_at] = needle
    return "\n".join(lines)


def prompt_with_needle(n_lines, where, code="MANGO-4417"):
    """where is a fraction from 0.0 (very start) to 1.0 (very end)."""
    needle = f"The secret passphrase is {code}."
    at = min(n_lines - 1, int(where * (n_lines - 1)))
    return haystack(n_lines, needle, at) + "\n\n" + QUESTION

Ask the same question with two window sizes

The next script builds a 300-line prompt with the fact in the first line and asks for the passphrase twice: once with the default window and once with num_ctx set to 16,384 in the request’s options. Then it repeats the default-window question for prompts of different lengths. To print the true size of each prompt it uses a trick that Step 5 explains: it asks Ollama to refuse the prompt and reads the token count out of the refusal.

# step02_reproduce.py
"""Plant a fact at the start of a long prompt and see whether the model can still read it."""
from olc import chat, too_big
from haystack import prompt_with_needle

MODEL = "qwen2.5:1.5b"


def ask(prompt, **kw):
    return chat(MODEL, [{"role": "user", "content": prompt}], **kw)


def real_tokens(prompt):
    """The true prompt size: ask Ollama to refuse (truncate=false) and read the number it reports."""
    status, data = ask(prompt, truncate=False)
    if status == 200:
        return data["prompt_eval_count"]
    return too_big(status, data)["n_prompt_tokens"]


prompt = prompt_with_needle(300, 0.0)
print(f"prompt: {len(prompt):,} characters, fact planted in the first line")
for label, options in [("default context", {}), ("num_ctx 16384", {"num_ctx": 16384})]:
    status, data = ask(prompt, options=options)
    print(f"{label:<16} HTTP {status}  prompt_eval_count={data['prompt_eval_count']:<5} done_reason={data['done_reason']}  answer={data['message']['content'].strip()!r}")

print()
print("lines  real tokens  prompt_eval_count  found the fact?")
for lines in [100, 140, 145, 150, 160, 200, 300, 450, 600]:
    prompt = prompt_with_needle(lines, 0.0)
    status, data = ask(prompt)
    found = "MANGO-4417" in data["message"]["content"]
    print(f"{lines:<6} {real_tokens(prompt):<12,} {data['prompt_eval_count']:<18,} {'yes' if found else 'NO'}")

print()
print("window   prompt_eval_count for the same 8,427-token prompt")
prompt = prompt_with_needle(300, 0.0)
for ctx in [2048, 4096, 8192]:
    status, data = ask(prompt, options={"num_ctx": ctx})
    print(f"{ctx:<8} {data['prompt_eval_count']:,}")
prompt: 16,257 characters, fact planted in the first line
default context  HTTP 200  prompt_eval_count=2050  done_reason=stop  answer='None provided in the log lines.'
num_ctx 16384    HTTP 200  prompt_eval_count=8427  done_reason=stop  answer='MANGO-4417'

That is the failure in two lines. The default window answered “None provided in the log lines.” with HTTP 200 and done_reason=stop, as if everything were fine, while the larger window found MANGO-4417. Look at prompt_eval_count: the prompt is 8,427 tokens, and with the default window the model processed 2,050. Now the table that shows where the cliff is:

lines  real tokens  prompt_eval_count  found the fact?
100    2,827        2,827              yes
140    3,947        3,947              yes
145    4,087        4,087              yes
150    4,227        2,050              NO
160    4,507        2,050              NO
200    5,627        2,050              NO
300    8,427        2,050              NO
450    12,627       2,050              NO
600    16,827       2,050              NO

Up to 4,087 tokens the prompt passes through untouched and the count is exact. At 4,227 tokens, which is only 131 over the window, the count does not become 4,096. It drops to 2,050, and it stays at 2,050 for every longer prompt, from 4,507 tokens up to 16,827. So the damage is not proportional: a prompt that is a little too long loses about half of itself. The last small table explains the number. With three different windows the same oversized prompt produced:

window   prompt_eval_count for the same 8,427-token prompt
2048     1,026
4096     2,050
8192     4,098

In these runs the processed count was always half the window plus two. I did not read the code that does this, so treat “half the window plus two” as a measured rule for this model and this version, not a documented one. The practical meaning is the part to remember: when you see prompt_eval_count far below your real prompt size, text was dropped.

Common mistake: blaming the model. A 1.5 billion parameter model is weak, but in this test the 16,384 window shows it can read the fact perfectly well. The same model with the same question fails only when the text never arrives.

Step 3: Map what survives

Knowing that half a window survives, the next question is which half. The script below moves the planted fact through the same 8,427-token prompt in ten percent steps and asks for it each time, with the default window and with a 16,384 window as the control.

# step03_survival_map.py
"""Move the planted fact through an 8,427-token prompt and see which positions survive."""
from olc import chat
from haystack import prompt_with_needle

MODEL = "qwen2.5:1.5b"
print("position in prompt   default (4096)   num_ctx 16384")
for position in [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]:
    prompt = prompt_with_needle(300, position)
    cells = []
    for options in ({}, {"num_ctx": 16384}):
        status, data = chat(MODEL, [{"role": "user", "content": prompt}], options=options)
        cells.append("found" if "MANGO-4417" in data["message"]["content"] else "LOST")
    print(f"{position:>6.0%}               {cells[0]:<15}  {cells[1]}")
position in prompt   default (4096)   num_ctx 16384
    0%               LOST             found
   10%               LOST             found
   20%               LOST             found
   30%               LOST             found
   40%               LOST             found
   50%               LOST             found
   60%               LOST             found
   70%               LOST             found
   80%               found            found
   90%               found            found
  100%               found            found

The control column says found in every row, so the model can read the fact anywhere when it can see it. The default column says LOST for every position up to 70 percent and found from 80 percent on. The cut keeps the end of the prompt and discards the beginning. The sizes agree: 20 percent of 8,427 tokens is about 1,700 tokens, which fits inside the 2,050 that survive, while 30 percent is about 2,500, which does not.

This has a practical consequence that is easy to miss. Instructions usually sit at the top of a prompt, because that is where people write them. Documents and chat history pile up below. A cut that keeps the bottom and drops the top removes exactly the part you wrote with the most care. Putting the question last, as this lab does, saves the question but not your instructions. This is one model and one prompt shape, so run the script with your own typical prompt before relying on the exact boundary.

Step 4: The same cut through other doors

Ollama has more than one place where text can be cut. The cut in Step 2 hit a single huge message, so it was made further down, after the prompt had been turned into tokens. For chats, the server adds a separate layer on top, and its source code explains the rule in a comment: “chatPrompt truncates any messages that exceed the context window of the model, making sure to always include 1) the latest message and 2) system messages”. In words: when a conversation is too long, the server drops the oldest messages first, keeps the system message and the newest message, and sends the rest on. If even that is too big, the lower layer cuts as in Step 2. The next script tests four ways in.

# step04_other_doors.py
"""The same silent cut through four other doors: chat history, a system message, the OpenAI-style endpoint, /api/generate."""
from olc import call, chat
from haystack import haystack, prompt_with_needle

MODEL = "qwen2.5:1.5b"


def show(label, status, data):
    if status != 200:
        print(f"{label:<28} HTTP {status}")
        return
    print(f"{label:<28} prompt_eval_count={data['prompt_eval_count']:<5} answer={data['message']['content'].strip()[:44]!r}")


print("1. A 40-turn chat; the codename is told in turn 1 and asked about in turn 41")
history = []
for turn in range(40):
    text = haystack(6, seed=100 + turn)
    if turn == 0:
        text = "My project codename is BLUEHERON. Remember it." + chr(10) + text
    history += [{"role": "user", "content": text}, {"role": "assistant", "content": "Noted."}]
history.append({"role": "user", "content": "What is my project codename? Answer with the codename only."})
show("default context", *chat(MODEL, history))
show("num_ctx 16384", *chat(MODEL, history, options={"num_ctx": 16384}))

print()
print("2. A system message with a fact, then one giant user message")
system = {"role": "system", "content": "Company policy: the refund window is 45 days."}
giant = haystack(300) + chr(10) * 2 + "Question: what is the refund window in days? Answer with the number only."
show("default context", *chat(MODEL, [system, {"role": "user", "content": giant}]))
show("num_ctx 16384", *chat(MODEL, [system, {"role": "user", "content": giant}], options={"num_ctx": 16384}))

print()
print("3. The OpenAI-compatible endpoint and 4. /api/generate, fact in the first line")
prompt = prompt_with_needle(300, 0.0)
status, data = call("/v1/chat/completions", {"model": MODEL, "messages": [{"role": "user", "content": prompt}], "temperature": 0, "seed": 1})
print(f"{'/v1/chat/completions':<28} usage.prompt_tokens={data['usage']['prompt_tokens']:<5} answer={data['choices'][0]['message']['content'][:44]!r}")
status, data = call("/api/generate", {"model": MODEL, "prompt": prompt, "stream": False, "options": {"temperature": 0, "seed": 1}})
print(f"{'/api/generate':<28} prompt_eval_count={data['prompt_eval_count']:<5} answer={data['response'].strip()[:44]!r}")
1. A 40-turn chat; the codename is told in turn 1 and asked about in turn 41
default context              prompt_eval_count=4033  answer='Project Phoenix'
num_ctx 16384                prompt_eval_count=7295  answer='BLUEHERON'

2. A system message with a fact, then one giant user message
default context              prompt_eval_count=2050  answer='5'
num_ctx 16384                prompt_eval_count=8441  answer='45'

3. The OpenAI-compatible endpoint and 4. /api/generate, fact in the first line
/v1/chat/completions         usage.prompt_tokens=2050  answer='None provided in the log lines.'
/api/generate                prompt_eval_count=2050  answer='None provided in the log lines.'

Read the four results one at a time. In the 40-turn chat, the codename told in the first turn was dropped as an old message. The prompt shrank to 4,033 tokens, just under the window, and the model, which had nothing to go on, made up “Project Phoenix” instead of saying it did not know. With a larger window the same conversation produced BLUEHERON. In the second test, a fact that lives in the system message, the company’s 45-day refund window, was lost because one giant user message pushed the whole prompt over the limit, and the cut removed the front of the prompt, which is where the system message sits. The model answered “5”. This is the case that matters for safety: a rule or a fact you place in the system message is not protected from a large enough user message. If a user, or a pasted document, can make the user message bigger than the window, the rules you wrote may never reach the model. The 45-day fact is a harmless stand-in, so replace it with a canary instruction of your own and test it the same way.

The last two lines show that the OpenAI-compatible endpoint and /api/generate behave the same way: the OpenAI-style response reports usage.prompt_tokens=2050 and the wrong answer. That endpoint has no options field, so you cannot raise the window per request there. Ollama’s documentation says so directly: “The OpenAI API does not have a way of setting the context size for a model.” For that endpoint you need one of the server-wide fixes from Step 6.

Step 5: Make Ollama refuse instead of truncate

Raising the window fixes the example prompts, but a bigger window is still finite, and real inputs keep growing. What you want is for an oversized prompt to fail loudly so your code can decide what to do. Ollama 0.32.9 has two request fields for this. In the chat and generate requests they are documented in the API types of its source code rather than on the API reference page I read. truncate: “Truncate is a boolean that, when set to true, truncates the chat history messages if the rendered prompt exceeds the context length limit.” shift: “Shift is a boolean that, when set to true, shifts the chat history when hitting the context length limit instead of erroring.” Both default to on for this model. Set them to false and see what happens.

# step05_fail_loud.py
"""Turn the silent cut into an error you can catch."""
from olc import chat, too_big
from haystack import prompt_with_needle

MODEL = "qwen2.5:1.5b"
messages = [{"role": "user", "content": prompt_with_needle(300, 0.0)}]
for label, flags in [("truncate=false", {"truncate": False}), ("shift=false", {"shift": False}), ("both false", {"truncate": False, "shift": False})]:
    status, data = chat(MODEL, messages, **flags)
    print(f"{label}: HTTP {status}")
    print("  ", data["error"])

print()
status, data = chat(MODEL, messages, truncate=False)
details = too_big(status, data)
print("parsed from the error string:", {key: details[key] for key in ("type", "n_prompt_tokens", "n_ctx")})
status, data = chat(MODEL, messages, options={"num_ctx": 16384}, truncate=False, shift=False)
print(f"same flags with num_ctx 16384: HTTP {status}, prompt_eval_count={data['prompt_eval_count']}, answer={data['message']['content'].strip()!r}")

print()
print("When the window fills during the answer (a prompt of about 3,960 tokens, num_ctx 4096, num_predict 600):")
long_ask = prompt_with_needle(140, 0.5).replace("Answer with the passphrase only.", "Then print the numbers 1 to 600, one per line, never abbreviate.")
for label, flags in [("default", {}), ("truncate=false, shift=false", {"truncate": False, "shift": False})]:
    status, data = chat(MODEL, [{"role": "user", "content": long_ask}], options={"num_ctx": 4096, "num_predict": 600}, **flags)
    print(f"{label:<28} HTTP {status}  prompt_eval_count={data['prompt_eval_count']}  eval_count={data['eval_count']}  done_reason={data['done_reason']}")
truncate=false: HTTP 400
   {"error":{"code":400,"message":"request (8427 tokens) exceeds the available context size (4096 tokens), try increasing it","type":"exceed_context_size_error","n_prompt_tokens":8427,"n_ctx":4096}}
shift=false: HTTP 400
   the prompt is longer than the context length currently available to the model; shorten the prompt, adjust the context length in settings, or use a model with a longer context length
both false: HTTP 400
   {"error":{"code":400,"message":"request (8427 tokens) exceeds the available context size (4096 tokens), try increasing it","type":"exceed_context_size_error","n_prompt_tokens":8427,"n_ctx":4096}}

Each flag alone, and both together, turned the silent cut into an HTTP 400 error, which is what we wanted. Notice the odd shape of two of the answers: Ollama passes the underlying JSON error through as a string inside its own error field. That is why too_big in olc.py parses it a second time. The flag shift=false alone produces a plain sentence instead, with no numbers, so the guard in Step 8 sets both flags and reads the structured form.

The refusal carries the exact token count and window size, and it comes back without the model reading your text (Step 8 times it), so once the model is loaded it works as a nearly free token counter:

parsed from the error string: {'type': 'exceed_context_size_error', 'n_prompt_tokens': 8427, 'n_ctx': 4096}
same flags with num_ctx 16384: HTTP 200, prompt_eval_count=8427, answer='MANGO-4417'

The second line shows the same flags are harmless when the window is big enough: the request succeeds and the model finds the fact. The last part of the output covers a case that surprises people. The window holds your prompt and the model’s answer together, so even a prompt that fits can leave too little room to answer:

When the window fills during the answer (a prompt of about 3,960 tokens, num_ctx 4096, num_predict 600):
default                      HTTP 200  prompt_eval_count=3961  eval_count=600  done_reason=length
truncate=false, shift=false  HTTP 200  prompt_eval_count=3961  eval_count=135  done_reason=length

With the defaults, the answer ran to the 600 tokens we allowed, because Ollama slides the window forward while it generates. With truncate=false, shift=false the answer stopped after 135 tokens, and 3,961 plus 135 is exactly 4,096. The reply is still HTTP 200 and done_reason=length in both rows, so the only clue is that eval_count (135) is smaller than the 600 tokens you allowed. The guard in Step 8 treats that pattern as “the window ran out” and retries with a bigger one.

Step 6: Set the window three ways, and prove each one

You can set num_ctx in three places. They differ in how far they reach, so it is worth knowing all three. The script after the list of ways tries each one against a prompt of 4,227 tokens, which is just over the default window, and asks /api/ps what was loaded.

Way 1: in the request

Put num_ctx in the options of a single request, as Step 2 did. It affects only that call, which makes it the safest way to experiment. The catch is that the window is a property of the loaded model, so a request that asks for a different size makes Ollama reload the model. The output below measures it.

Way 2: in a Modelfile

A Modelfile is a short recipe for a named model. Add the line PARAMETER num_ctx 8192 and every request to that model name gets the window with no options needed. This is also the fix for clients that cannot send options, such as anything that talks to the OpenAI-compatible endpoint. The documentation for that endpoint says: “If you need to change the context size, create a Modelfile”. The script creates a model called qwen2.5-1.5b-8k from a Modelfile and removes it again at the end.

Way 3: for the whole server

The environment variable OLLAMA_CONTEXT_LENGTH changes the default for every model the server loads. The FAQ gives the shell form, OLLAMA_CONTEXT_LENGTH=8192 ollama serve. In Windows PowerShell the same thing is $env:OLLAMA_CONTEXT_LENGTH = "8192"; ollama serve. If the Ollama desktop app starts your server, set the variable as the FAQ describes for Windows (“Ollama inherits your user and system environment variables”, so quit Ollama, edit your user environment variables, and start it again), or use the context length slider in the app’s settings that the context length page describes. To keep your running server untouched, the script starts a second server on port 11435 with the variable set and stops it afterwards.

# step06_set_it.py
"""Three ways to set the context window, each proven with /api/ps."""
import os
import subprocess
import tempfile
import time

from olc import HOST, call, chat, loaded, unload
from haystack import prompt_with_needle

MODEL = "qwen2.5:1.5b"
CUSTOM = "qwen2.5-1.5b-8k"
SECOND = "http://127.0.0.1:11435"
messages = [{"role": "user", "content": prompt_with_needle(150, 0.0)}]  # 4,227 tokens: just over the 4096 default


def report(label, host=HOST, model=MODEL, **kw):
    status, data = chat(model, messages, host, **kw)
    window = [m["context_length"] for m in loaded(host)]
    found = "MANGO-4417" in data["message"]["content"]
    load = data["load_duration"] / 1e6
    print(f"{label:<30} loaded context={window}  prompt_eval_count={data['prompt_eval_count']:<5} fact found={found}  model load {load:.0f} ms")


unload(MODEL)
report("nothing set")
unload(MODEL)
report("1. options.num_ctx = 8192", options={"num_ctx": 8192})
report("   same request again", options={"num_ctx": 8192})
report("   num_ctx = 16384, no unload", options={"num_ctx": 16384})
unload(MODEL)

folder = tempfile.mkdtemp()
modelfile = os.path.join(folder, "Modelfile")
with open(modelfile, "w", newline="\n") as handle:
    handle.write(f"FROM {MODEL}\nPARAMETER num_ctx 8192\n")
subprocess.run(["ollama", "create", CUSTOM, "-f", modelfile], check=True, capture_output=True)
unload(MODEL)
report("2. Modelfile num_ctx 8192", model=CUSTOM)
print("   ollama show --parameters:", subprocess.run(["ollama", "show", CUSTOM, "--parameters"], capture_output=True, text=True).stdout.split())
unload(CUSTOM)
subprocess.run(["ollama", "rm", CUSTOM], check=True, capture_output=True)

env = dict(os.environ, OLLAMA_HOST="127.0.0.1:11435", OLLAMA_CONTEXT_LENGTH="8192")
server = subprocess.Popen(["ollama", "serve"], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
    for _ in range(40):
        try:
            call("/api/version", host=SECOND, timeout=2)
            break
        except OSError:
            time.sleep(0.5)
    report("3. OLLAMA_CONTEXT_LENGTH=8192", host=SECOND)
    unload(MODEL, SECOND)
finally:
    subprocess.run(["taskkill", "/PID", str(server.pid), "/T", "/F"], capture_output=True)
nothing set                    loaded context=[4096]  prompt_eval_count=2050  fact found=False  model load 1781 ms
1. options.num_ctx = 8192      loaded context=[8192]  prompt_eval_count=4227  fact found=True  model load 1767 ms
   same request again          loaded context=[8192]  prompt_eval_count=4227  fact found=True  model load 144 ms
   num_ctx = 16384, no unload  loaded context=[16384]  prompt_eval_count=4227  fact found=True  model load 2582 ms
2. Modelfile num_ctx 8192      loaded context=[8192]  prompt_eval_count=4227  fact found=True  model load 2299 ms
   ollama show --parameters: ['num_ctx', '8192']
3. OLLAMA_CONTEXT_LENGTH=8192  loaded context=[8192]  prompt_eval_count=4227  fact found=True  model load 1843 ms

Read the output from the top. With nothing set, the window is 4,096, the prompt is cut to 2,050 tokens, and the fact is not found. Way 1 loads an 8,192 window, the model processes all 4,227 tokens, and the fact is found. The last column is the important teaching detail: loading the model took 1,767 ms the first time, 144 ms for the same request again, and 2,582 ms when the next request asked for 16,384 without unloading first. A new window means a reload. If two programs share one Ollama server and ask for different sizes, they will make the model reload each time they take turns, so agree on one window per model. Way 2 prints num_ctx 8192 from ollama show --parameters, and Way 3 gives the same 8,192 window with no option anywhere in the request.

How to check this step worked: ollama ps shows your chosen number in the CONTEXT column while the model is loaded. If you set the environment variable and it still says 4096, the server you are talking to was started somewhere that did not see the variable. Set it where the server starts and restart it.

Step 7: Price the window before you raise it

A bigger window is not free. The model keeps a key and a value for every token in the window, in every layer, and that memory is reserved up front. The FAQ warns about the same effect from the other side: “Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH”, so the number of parallel requests multiplies it. The context length page puts the rule in one sentence: “Setting a larger context length will increase the amount of memory required to run a model.”

You can predict the cache size with arithmetic. Each token needs two numbers (a key and a value) per layer for each of the model’s key-value heads, and each number takes two bytes at the default 16-bit precision. The script reads the layer count and head sizes from /api/show, predicts the growth, and compares it with the size Ollama reports after loading the model at each window. It uses a prompt of about 1,550 tokens so that every row fits, asks for a single output token, and finally repeats the largest window with the cache quantized to 8 bits on a second server.

# step07_cost.py
"""What a bigger context window costs in memory and time."""
import os
import subprocess
import time

from olc import HOST, call, chat, loaded, unload
from haystack import prompt_with_needle

MODEL = "qwen2.5:1.5b"
SECOND = "http://127.0.0.1:11435"
info = call("/api/show", {"model": MODEL})[1]["model_info"]
layers = info["qwen2.block_count"]
kv_heads = info["qwen2.attention.head_count_kv"]
head_dim = info["qwen2.embedding_length"] // info["qwen2.attention.head_count"]
per_token = 2 * layers * kv_heads * head_dim * 2  # K and V, 2 bytes per number in f16
print(f"layers={layers}, kv_heads={kv_heads}, head_dim={head_dim}: {per_token:,} bytes of KV cache per token")

messages = [{"role": "user", "content": prompt_with_needle(55, 0.0)}]  # about 1,550 tokens, fits every row


def measure(ctx, host=HOST):
    unload(MODEL, host)
    status, data = chat(MODEL, messages, host, options={"num_ctx": ctx, "num_predict": 1})
    entry = loaded(host)[0]
    speed = data["prompt_eval_count"] / (data["prompt_eval_duration"] / 1e9)
    return entry["context_length"], entry["size"] / 2**20, data["load_duration"] / 1e6, speed


print()
print("asked     loaded      loaded size   growth vs 2048   KV arithmetic   load time   prefill")
first = None
for ctx in [2048, 4096, 8192, 16384, 32768, 65536]:
    window, size, load_ms, speed = measure(ctx)
    first = first or size
    arithmetic = (window - 2048) * per_token / 2**20
    print(f"{ctx:<9} {window:<11} {size:>7.0f} MiB   {size - first:>8.0f} MiB     {arithmetic:>8.0f} MiB    {load_ms:>6.0f} ms   {speed:>5.0f} tok/s")
unload(MODEL)

env = dict(os.environ, OLLAMA_HOST="127.0.0.1:11435", OLLAMA_FLASH_ATTENTION="1", OLLAMA_KV_CACHE_TYPE="q8_0")
server = subprocess.Popen(["ollama", "serve"], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
    for _ in range(40):
        try:
            call("/api/version", host=SECOND, timeout=2)
            break
        except OSError:
            time.sleep(0.5)
    window, size, load_ms, speed = measure(32768, SECOND)
    print()
    print(f"32768 with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0: {size:.0f} MiB, load {load_ms:.0f} ms, prefill {speed:.0f} tok/s")
    unload(MODEL, SECOND)
finally:
    subprocess.run(["taskkill", "/PID", str(server.pid), "/T", "/F"], capture_output=True)
layers=28, kv_heads=2, head_dim=128: 28,672 bytes of KV cache per token

asked     loaded      loaded size   growth vs 2048   KV arithmetic   load time   prefill
2048      2048           1054 MiB          0 MiB            0 MiB      1757 ms    2000 tok/s
4096      4096           1112 MiB         58 MiB           56 MiB      1762 ms    2000 tok/s
8192      8192           1298 MiB        244 MiB          168 MiB      1749 ms    2003 tok/s
16384     16384          1538 MiB        484 MiB          392 MiB      1777 ms    1976 tok/s
32768     32768          2018 MiB        964 MiB          840 MiB      1755 ms    2007 tok/s
65536     32768          2018 MiB        964 MiB          840 MiB      1739 ms    2002 tok/s

32768 with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0: 1598 MiB, load 1830 ms, prefill 1939 tok/s

This model has 28 layers, 2 key-value heads and a head size of 128, which makes 2 × 28 × 2 × 128 × 2 = 28,672 bytes of cache per token. From a 2,048 window to a 32,768 window the loaded size grew by 964 MiB. The arithmetic predicts 840 MiB for the cache alone, so the measured growth is higher in every row, by 2 to 124 MiB. I did not break down the difference; other buffers that also scale with the window are the likely reason. The size almost doubled, from 1,054 to 2,018 MiB, while the model file itself is under 1 GB. On this card the load time (about 1.75 seconds) and the prefill speed (about 2,000 tokens per second) did not change from row to row for this short prompt, so the price here is memory. I did not measure a prompt that fills a large window, where the work per token grows, so measure that with your real prompts.

Two more lines in the table deserve attention. Asking for 65,536 loaded 32,768, the model’s trained window, at exactly the same size as the row above, so a request for more than the model was trained for is lowered without any message. And the last line shows what the cache type buys you. The FAQ describes the setting: “The K/V context cache can be quantized to significantly reduce memory usage when Flash Attention is enabled.” With OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, the 32,768 window loaded at 1,598 MiB instead of 2,018 MiB, a saving of 420 MiB. The arithmetic agrees: a 32,768-token cache at 2 bytes per number is 896 MiB, the 8-bit format stores 32 numbers in 34 bytes (a 2-byte scale plus 32 one-byte values, as I understand ggml’s q8_0 layout), so it needs about 476 MiB, and 896 minus 476 leaves a saving of 420 MiB. The FAQ says q8_0 “uses approximately 1/2 the memory of f16 with a very small loss in precision”. That is a setting for the whole server, so test your model’s answers before adopting it.

That gives you a way to choose a window. Count the tokens in your longest realistic prompt (Step 5 gives the exact number), add the longest answer you will allow, add a margin, and round up to a standard size. Then load the model and read ollama ps. The context length page advises: “For best performance, use the maximum context length for a model, and avoid offloading the model to CPU. Verify the split under PROCESSOR using ollama ps.” If PROCESSOR shows part of the model on the CPU, your window is too big for your card, and lowering it is usually the better trade than accepting the slowdown. The same page says tasks such as agents and coding tools should use at least 64,000 tokens, which a model trained for 32,768 cannot offer, so that advice only applies to models with a longer trained window.

Step 8: Wrap it in a guard you can test

Steps 5 to 7 give you every piece of a safe call: refuse oversized prompts, read the exact size from the refusal, choose a window that fits, and notice when the window ran out during an answer. This step assembles them into one function so that no caller has to remember the rules. The design is deliberately small:

The guard starts at a cheap 4,096 window and always sends truncate=false and shift=false, so Ollama can only answer in full or refuse. If it refuses, the guard reads the prompt size from the refusal, adds the tokens you reserved for the answer, and picks the smallest standard window that holds both. If that window is above your budget (max_ctx), it raises PromptTooLong before any generation work is done. If a reply comes back with done_reason=length but fewer tokens than you allowed, the window filled during the answer, and the guard retries with a bigger one. Any other failure raises OllamaError. It tries at most three times.

# ctxguard.py
"""Call Ollama so that a prompt is either read in full or refused. Never cut silently."""
from olc import HOST, chat, too_big

TIERS = (2048, 4096, 8192, 16384, 32768, 65536, 131072)


class PromptTooLong(Exception):
    def __init__(self, tokens, budget):
        super().__init__(f"prompt is {tokens} tokens but the budget is {budget}")
        self.tokens = tokens
        self.budget = budget


class OllamaError(Exception):
    pass


def pick_context(needed, max_ctx):
    """The smallest standard window that holds `needed` tokens, or None if it would pass max_ctx."""
    for tier in TIERS:
        if tier >= needed:
            return tier if tier <= max_ctx else None
    return None


def window_ran_out(status, data, answer_tokens):
    """A reply that stops early with done_reason 'length' ended because the window filled, not because of num_predict."""
    return status == 200 and data.get("done_reason") == "length" and data.get("eval_count", 0) < answer_tokens


def safe_chat(model, messages, *, max_ctx=16384, answer_tokens=512, start_ctx=4096, host=HOST, options=None):
    """Return a complete reply, or raise. Sizes num_ctx from what Ollama reports; never lets it truncate or shift."""
    base = {**(options or {}), "num_predict": answer_tokens}
    ctx = start_ctx
    for _ in range(3):
        status, data = chat(model, messages, host, options={**base, "num_ctx": ctx}, truncate=False, shift=False)
        refused = too_big(status, data)
        if refused:
            prompt_tokens = refused["n_prompt_tokens"]
        elif window_ran_out(status, data, answer_tokens):
            prompt_tokens = data["prompt_eval_count"]
        elif status == 200:
            return data
        else:
            raise OllamaError(f"HTTP {status}: {data.get('error')}")
        ctx = pick_context(prompt_tokens + answer_tokens, max_ctx)
        if ctx is None:
            raise PromptTooLong(prompt_tokens, max_ctx - answer_tokens)
    raise OllamaError("could not size the context window in three tries")

Test it without a model

You do not need a language model to test this logic, only something that answers like Ollama does. The test file below starts a tiny fake server in a thread. It charges 20 tokens per message plus one token per word, refuses oversized prompts the way the real server does (JSON inside a string), and silently cuts them to half a window plus two when the flags are missing. That last detail matters: it means a guard that forgets the flags gets a wrong answer from the fake server, and a test can notice. Each test checks one rule from the design above.

# test_ctxguard.py
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import pytest

import ctxguard


class FakeOllama:
    """Pretends to be /api/chat. A message of N words costs N + 20 tokens; the model wants to write answer_tokens tokens."""

    def __init__(self):
        self.calls = []
        self.answer_tokens = 30
        self.fail_with = None
        outer = self

        class Handler(BaseHTTPRequestHandler):
            def log_message(self, *args):
                pass

            def do_POST(self):
                body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
                status, payload = outer.respond(body)
                raw = json.dumps(payload).encode()
                self.send_response(status)
                self.send_header("Content-Type", "application/json")
                self.send_header("Content-Length", str(len(raw)))
                self.end_headers()
                self.wfile.write(raw)

        self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
        self.url = f"http://127.0.0.1:{self.server.server_address[1]}"
        threading.Thread(target=self.server.serve_forever, daemon=True).start()

    def respond(self, body):
        options = body["options"]
        ctx = options["num_ctx"]
        self.calls.append({"num_ctx": ctx, "truncate": body.get("truncate"), "shift": body.get("shift"), "temperature": options.get("temperature")})
        if self.fail_with:
            return self.fail_with, {"error": "boom"}
        tokens = sum(len(m["content"].split()) + 20 for m in body["messages"])
        strict = body.get("truncate") is False and body.get("shift") is False
        if tokens > ctx:
            if strict:
                refusal = {"error": {"code": 400, "message": "too big", "type": "exceed_context_size_error", "n_prompt_tokens": tokens, "n_ctx": ctx}}
                return 400, {"error": json.dumps(refusal)}  # like the real server: JSON inside a string
            return 200, {"message": {"content": "ok"}, "prompt_eval_count": ctx // 2 + 2, "eval_count": 5, "done_reason": "stop"}
        used = min(self.answer_tokens, options["num_predict"], ctx - tokens)
        return 200, {"message": {"content": "ok"}, "prompt_eval_count": tokens, "eval_count": used, "done_reason": "stop" if used == self.answer_tokens else "length"}


@pytest.fixture
def fake():
    return FakeOllama()


def words(count):
    return [{"role": "user", "content": " ".join(["word"] * count)}]


def run(fake, count, **kw):
    return ctxguard.safe_chat("m", words(count), host=fake.url, **kw)


def test_short_prompt_takes_one_call_at_the_start_window(fake):
    data = run(fake, 100)
    assert data["done_reason"] == "stop"
    assert [c["num_ctx"] for c in fake.calls] == [4096]


def test_big_prompt_is_refused_then_sized_from_the_reported_count(fake):
    data = run(fake, 6000)  # 6,020 tokens plus 512 for the answer needs 8192
    assert [c["num_ctx"] for c in fake.calls] == [4096, 8192]
    assert data["prompt_eval_count"] == 6020


def test_prompt_over_budget_raises_after_one_cheap_call(fake):
    with pytest.raises(ctxguard.PromptTooLong) as caught:
        run(fake, 40000, max_ctx=16384)
    assert caught.value.tokens == 40020
    assert len(fake.calls) == 1


def test_every_call_forbids_truncating_and_shifting(fake):
    run(fake, 6000)
    assert all(c["truncate"] is False and c["shift"] is False for c in fake.calls)


def test_window_that_fills_during_the_answer_is_retried_bigger(fake):
    fake.answer_tokens = 400
    data = run(fake, 3800, answer_tokens=512)  # 3,820 prompt tokens leave 276 of 4096
    assert [c["num_ctx"] for c in fake.calls] == [4096, 8192]
    assert data["done_reason"] == "stop" and data["eval_count"] == 400


def test_unexpected_server_error_is_raised_not_swallowed(fake):
    fake.fail_with = 500
    with pytest.raises(ctxguard.OllamaError):
        run(fake, 10)


def test_other_400_errors_are_not_mistaken_for_size_refusals(fake):
    fake.fail_with = 400
    with pytest.raises(ctxguard.OllamaError):
        run(fake, 10)
    assert len(fake.calls) == 1


def test_callers_options_still_reach_the_server(fake):
    run(fake, 10, options={"temperature": 0.3})
    assert fake.calls[0]["temperature"] == 0.3


@pytest.mark.parametrize(
    "needed, max_ctx, expected",
    [(2048, 131072, 2048), (2049, 131072, 4096), (4096, 131072, 4096), (4097, 16384, 8192), (16385, 16384, None), (200000, 10**9, None)],
)
def test_pick_context_boundaries(needed, max_ctx, expected):
    assert ctxguard.pick_context(needed, max_ctx) == expected
..............                                                                                                                                                     [100%]
14 passed in 0.09s

Check that the tests would catch a bug

Passing tests prove little if they cannot fail. The script below plants eight bugs in ctxguard.py, one at a time, runs the tests against each, and restores the file. Every bug should make at least one test fail.

# mutate.py
"""Plant one bug at a time in ctxguard.py and check that the tests notice."""
import os
import pathlib
import shutil
import subprocess
import sys

SOURCE = pathlib.Path("ctxguard.py")
MUTANTS = [
    ("allow truncation", "truncate=False, shift=False)", "truncate=True, shift=False)"),
    ("allow shifting", "truncate=False, shift=False)", "truncate=False, shift=True)"),
    ("tier boundary off by one", "if tier >= needed:", "if tier > needed:"),
    ("ignore the answer budget", "pick_context(prompt_tokens + answer_tokens, max_ctx)", "pick_context(prompt_tokens, max_ctx)"),
    ("ignore max_ctx", "return tier if tier <= max_ctx else None", "return tier"),
    ("trust a reply that ran out of window", "elif window_ran_out(status, data, answer_tokens):", "elif False:"),
    ("never recognise a size refusal", "if refused:", "if False:"),
    ("swallow server errors", 'raise OllamaError(f"HTTP', 'return data  # raise OllamaError(f"HTTP'),
]


def run_tests():
    shutil.rmtree("__pycache__", ignore_errors=True)
    env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", COLUMNS="170")
    result = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "test_ctxguard.py"],
        capture_output=True, text=True, env=env,
    )
    failed = [line for line in result.stdout.splitlines() if line.startswith("FAILED")]
    return result.returncode, len(failed)


original = SOURCE.read_text()
code, _ = run_tests()
print(f"clean source: tests exit code {code}")
caught = 0
try:
    for name, old, new in MUTANTS:
        assert original.count(old) == 1, name
        SOURCE.write_text(original.replace(old, new))
        code, failed = run_tests()
        caught += code != 0
        print(f"{name:<40} {failed} failing tests -> {'caught' if code else 'SURVIVED'}")
finally:
    SOURCE.write_text(original)
print(f"{caught} of {len(MUTANTS)} planted bugs caught")
clean source: tests exit code 0
allow truncation                         3 failing tests -> caught
allow shifting                           3 failing tests -> caught
tier boundary off by one                 2 failing tests -> caught
ignore the answer budget                 1 failing tests -> caught
ignore max_ctx                           2 failing tests -> caught
trust a reply that ran out of window     1 failing tests -> caught
never recognise a size refusal           3 failing tests -> caught
swallow server errors                    2 failing tests -> caught
8 of 8 planted bugs caught

All eight were caught. The two flag mutants fail three tests each, because the fake server then truncates silently and the guard’s answers go wrong. The mutants in the sizing code (the tier boundary, the answer reserve and the budget) fail only one or two tests each, which shows that each of those rules is pinned down by a test written for it. This is also how I know the window-filled test earns its place: without it, the “trust a reply that ran out of window” bug would survive.

Run it against the real server

Finally, three calls to the real model: a short question, the 8,427-token prompt from Step 2, and a prompt of about 25,000 tokens that is over the budget.

# step08_guard_live.py
"""Run the guard against the real server."""
import time

from ctxguard import PromptTooLong, safe_chat
from haystack import prompt_with_needle
from olc import loaded, unload

MODEL = "qwen2.5:1.5b"
unload(MODEL)


def ask(label, text):
    started = time.time()
    try:
        data = safe_chat(MODEL, [{"role": "user", "content": text}])
    except PromptTooLong as err:
        print(f"{label:<22} refused: {err} ({time.time() - started:.1f} s)")
        return
    window = [m["context_length"] for m in loaded()]
    print(f"{label:<22} window={window} prompt_eval_count={data['prompt_eval_count']:<6} answer={data['message']['content'].strip()[:30]!r} ({time.time() - started:.1f} s)")


ask("short question", "Name the capital of France in one word.")
ask("8,427-token prompt", prompt_with_needle(300, 0.0))
ask("25,000-token prompt", prompt_with_needle(900, 0.0))
ask("25,000 tokens, asked again", prompt_with_needle(900, 0.0))
short question         window=[4096] prompt_eval_count=38     answer='Paris.' (1.9 s)
8,427-token prompt     window=[16384] prompt_eval_count=8427   answer='MANGO-4417' (7.5 s)
25,000-token prompt    refused: prompt is 25227 tokens but the budget is 15872 (2.6 s)
25,000 tokens, asked again refused: prompt is 25227 tokens but the budget is 15872 (0.2 s)

The short question stayed at the 4,096 window and cost one call. The 8,427-token prompt was refused at 4,096, then answered correctly after the guard moved to 16,384 (8,427 plus 512 reserved tokens needs more than 8,192). The 25,000-token prompt was refused at once with a clear message, “prompt is 25227 tokens but the budget is 15872”, which is the 16,384 budget minus the 512 tokens reserved for the answer. That first refusal took 2.6 seconds, because the model had to reload at the small starting window. The fourth line asks the same question again with the model already loaded that way, and the refusal came back in 0.2 seconds, which is the real cost of a rejection: it arrives before the model reads your text.

Confirm everything works end to end

Run the full chain once more as a final check. In the lab folder, run python step02_reproduce.py and confirm you can reproduce the cliff on your own machine, with prompt_eval_count collapsing for prompts that are larger than your window. Run python -m pytest -q test_ctxguard.py and confirm all tests pass. Run python step08_guard_live.py and confirm the short question stays small, the large prompt gets a larger window and the right answer, and the oversized prompt is refused. Finally, call ollama ps and read the CONTEXT column once more. If those four checks agree with what this page printed, your setup can no longer cut a prompt silently.

Common mistakes and how to spot them

Setting the variable where the server cannot see it

You export OLLAMA_CONTEXT_LENGTH in one terminal, but the server was started by the desktop app or a service, so it never saw the variable. The tell is ollama ps still showing 4096. Set the variable where the server starts, restart the server, and check the CONTEXT column.

Trusting a bigger number than the model supports

Asking for 65,536 on a model trained for 32,768 gave a 32,768 window with no message, so a pipeline that assumes 65,536 will still be cut. Compare the window you asked for with the context_length in /api/ps after the first request.

Reading prompt_eval_count without knowing the real size

The count only exposes a cut if you know how many tokens you sent. Log it next to the exact size from the refusal trick, or use the guard so the comparison is not left to chance.

Letting two clients fight over one model

Different window sizes make the model reload, which Step 6 timed at about 2.6 seconds against 144 milliseconds for a repeat. Pick one window per model name and give each use case its own Modelfile name if they truly need different sizes.

Raising the window until the model spills onto the CPU

The context length page tells you to avoid offloading the model to the CPU and to check the PROCESSOR column. If you see a CPU share, lower the window or quantize the cache, as Step 7 showed, and test again.

Where to go next

Replace the log-line prompt with your own typical prompt and rerun Steps 2 and 3 to find where your cliff is. Add the guard to the code that calls Ollama and log every PromptTooLong, because each one is a prompt that used to be cut without anyone noticing. If you use Ollama through an OpenAI-compatible client, move the window into a Modelfile or the server setting and keep the guard’s native-API calls for checking. For related ground on this site, diagnosing a low prompt cache hit rate with Ollama covers what the server remembers between requests, benchmarking inference latency with Python and Ollama covers how to time what you change, and stopping PII leaks and prompt injection with guardrails covers rules that, as Step 4 showed, only help if they stay inside the window. Our agent harness tutorial ran into the same 2,050-token ceiling in a context probe.

Sources

The documentation quoted above, and the source files read for this lab, are all public:

  • Ollama documentation: Context length, FAQ, Modelfile reference and OpenAI compatibility.
  • Ollama source code at the v0.32.9 tag: server/prompt.go (message truncation), api/types.go (the truncate and shift fields) and server/routes.go (the video-memory default for the window).

Tags:

Context WindowKV CacheLLM InferenceLocal LLMOllamaPython

Share

A brown llama and her gray cria walking across a salt flat beside a red lake in Bolivia
Previous Post

llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
16 Nov
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
16 Nov
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
Trending
November 16, 2026
How to Stop Ollama From Silently Cutting Your Prompt, Find the Real Context Window, and Set num_ctx Without Running Out of Memory
November 16, 2026
llama.cpp’s v1.0.0 Promises Major Bumps for API and ABI Breaks, but Its Rule Table Lists Only the C API, libmtmd Shares the Version While Its Header Warns of Breaking Changes, and the Release Page Carries a 7-Byte Pointer to a Nightly Build Instead of Binaries
November 16, 2026
NoisyTCP Can Crash a Linux Server Over the Network, but the Fix Has Reached Only Three of the Seven Kernel Lines Still Being Released, Red Hat and Ubuntu Have No Update, and the Researcher’s Own Timings Run From 5 Seconds to 14 Hours
November 15, 2026
How to Route Alerts in Alertmanager So Pages Reach the On-Call and Tickets Reach the Queue, and Test Grouping, Inhibition, and Silences Before an Outage Does
November 15, 2026
Ai2’s XYBench Turns the XY Problem Into a Benchmark, but the Model That Drafted Every Rubric Also Judges the Answers, the Human Baseline Is Graded on Rubrics Read Off Its Own Answers, and the Emphasis Judge Disagrees With the Annotators on 20 to 26 Percent of Validation Responses
November 15, 2026
The FBI’s ARTEX Warning Says a Supply Chain Compromise “Enabled Access” to 2,250 Tenants Where Microsoft Counted Sign-In Attempts and Cites “Over 1,600” Consoles Where Censys Counted 1,465, and a Microsoft Post It Cites Mistypes One Indicator in Its Hunting Query

Related Posts

A laptop wrapped in a chain and padlock, illustrating least-privilege controls for AI agents.
Learning Hub

How to Secure Tool-Using AI Agents Before They Touch Production

June 8, 2026
Colorful sticky notes arranged on an office wall, symbolizing governance checklists and planning.
Learning Hub

AI Governance for Agentic Apps: A Practical Checklist for Builders

June 8, 2026
A technician connects green fiber optic cables at a data center, representing a private production inference endpoint.
Learning Hub

How to Deploy a Fine-Tuned LLM Behind a Private Production Inference Endpoint

June 8, 2026
Narrow aisle behind black supercomputer racks in a data center
Learning Hub

Kubernetes SELinux Volume Labeling: What Cluster Operators Should Audit Before v1.37

June 8, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026