TRENDING
A person sweeping a metal detector across a wet sandy beach with a sea fort on the horizon
November 22, 2026
How to Write YARA Rules With YARA-X to Hunt Malicious Python .pth Startup Files and Measure False Positives Before You Trust Them
A peeling black-and-white metal door standing ajar, with a dark empty hall and bright windows beyond
November 22, 2026
AWS’s “Found a Zero-Day on Their Own” Post Never Names the Product; the Advisory Behind Its Link Shows a Server That Ships With No Login, a README Now Saying Keep It Off the Internet, and a Fix Merged October 19 That PyPI and Docker Hub Still Do Not Carry
A hand paging through index cards in an open wooden library card-catalog drawer labeled Agriculture, with a second drawer open below to the right
November 22, 2026
Microsoft Tells Customers Unauthorized Parties Read Admin Email Addresses and Tenant Metadata Through an Entra Connect Health Flaw Introduced July 23, but Gives No Tenant Count or Start Date, and Its Own Docs Send Tenant Security Notices to the Same Kinds of Contacts
Swimmers in separate lanes of a blue swimming pool, seen from above the water, with lane lines running across the picture
November 22, 2026
How to Size a PostgreSQL Connection Pool in Python: Find Where Throughput Stops Growing, Split max_connections Across Workers, and Stop Slow Handlers From Hoarding Connections
Workers in caps lean over a 1913 Ford assembly line, each assembling parts as they pass on the moving line
November 22, 2026
OpenAI’s Performance Factory Turns Regression Response Into a Task Queue for Agents, but Its Numbers Stop at 600 Root-Caused Cases in 1,700 Investigations and a 70% Grade on 100 Suggestions, and in a Toy World Each of Its Reviewer’s Two Counterfactual Questions Can Return a Confident Wrong Answer
22 Nov 2026
SXZ.io SXZ.io
  • Home
Search the Site
Popular Searches:
Technology Amazon AI
Recent Posts
The sun setting over the sea behind a pyramid-shaped stone monument at Cape Soya, the northernmost point of Japan
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
November 22, 2026
Brass sextant in an open wooden case inside a museum display, an instrument for measuring the angle to the horizon
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
November 22, 2026
Brass telescope on a wooden tripod inside a white observatory dome with the roof slit open to the sky
Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”
November 22, 2026
SXZ.io SXZ.io
  • Home

Categories

Articles 310 Posts
News 316 Posts
Learning Hub 282 Posts
Home/Learning Hub/How to Serve Several Users From One Ollama Server Without Surprise Waits: Parallel Slots, the Queue Limit, keep_alive, and Model Swaps
Learning Hub

How to Serve Several Users From One Ollama Server Without Surprise Waits: Parallel Slots, the Queue Limit, keep_alive, and Model Swaps

Measure how Ollama 0.32.9 behaves when several people share one server, then set parallel slots, the queue limit, keep_alive and loaded models on purpose, with a tested Python gate that retries 503...

November 17, 2026 42 Min Read
47

Two people on your team point their scripts at the same Ollama server. The first one asks a short question and gets an answer in about two seconds. The second one asks at the same moment and waits nearly four. Add two more people and the last one stares at a blank screen for seven seconds, although the hardware had capacity to spare. Nothing is broken. Ollama is doing what its default settings tell it to do, and those defaults suit one person on one laptop. They are not a plan for a shared server.

Table Of Content

  • Before you start
  • What you need
  • A few words we will use
  • The files at a glance
  • Step 1: Read the settings the server really uses, and time one request
  • How Ollama takes its settings
  • Why the lab reads the server’s own log
  • What the output tells you
  • Step 2: Watch four users share one slot
  • The experiment
  • What happened
  • Step 3: Give the model more slots with OLLAMA_NUM_PARALLEL
  • What a slot buys you
  • Reading the table
  • The memory bill
  • Step 4: Find out when Ollama really answers 503
  • What the documentation promises
  • What happened
  • Why: the queue is a loading queue
  • What this means for you
  • Step 5: Control how long a model stays loaded with keep_alive
  • Why the timer matters
  • The experiment
  • Reading it row by row
  • The rule, and why the server behaves this way
  • What to do about it
  • Step 6: Keep more than one model ready with OLLAMA_MAX_LOADED_MODELS
  • The second level of concurrency
  • What happened
  • Choosing the number
  • Step 7: Spot a parallel setting that the server ignores
  • The setting that did nothing
  • What happened
  • How to catch it on your server
  • Step 8: Protect your own application with a small client-side gate
  • Three jobs for the gate
  • The code
  • Tests against a fake server
  • Do the tests really guard anything?
  • The gate against the real server
  • Reading the four rounds
  • Check that everything works end to end
  • Common mistakes
  • Where to go next
  • Sources

This tutorial teaches you to measure how an Ollama server behaves when several people use it at once, and then to choose four settings on purpose instead of by accident: how many requests one model works on in parallel (OLLAMA_NUM_PARALLEL), how many requests may wait in line (OLLAMA_MAX_QUEUE), how long a model stays in memory after its last request (keep_alive), and how many different models may be in memory together (OLLAMA_MAX_LOADED_MODELS). You will build a small lab, watch each setting change real numbers, and finish with a short Python gate that protects your own application from the rough edges, complete with tests.

Along the way you will meet three behaviors that the documentation does not spell out, each shown by an experiment and traced to the server’s source code: the queue limit does not apply to a model that is already loaded and busy, a keep-alive value stated by one request stays in force for everyone after it, and some models quietly ignore the parallel setting.

A note on scope. Every number below was measured on Ollama 0.32.9 on Windows 11 with an NVIDIA RTX A400 that has 4 GiB of video memory, using small models so that each step finishes in seconds. Your hardware will give other numbers. The shapes, the reasons behind them and the scripts that measure them carry over.

Before you start

What you need

You need Ollama 0.32.9 or a nearby version (check with ollama --version), two small models (ollama pull qwen2.5:1.5b and ollama pull qwen2.5:0.5b), and Python 3.13 or newer. Step 7 uses a third model, qwen3.5:4b, which is a download of a few gigabytes, so skip that step if you would rather not pull it. The lab uses only the Python standard library, plus pytest for the tests in Step 8. It was written to run on Windows, Linux and macOS, and I tested it on Windows 11.

The lab never touches your normal Ollama server. Every experiment starts its own throwaway ollama serve on a different local port, with exactly the environment variables that experiment needs, and stops it afterwards. Two servers share one GPU, so close anything else that is using it while the lab runs. You should be comfortable reading short Python scripts. No machine learning knowledge is assumed.

Put every file from this tutorial in one empty folder. The first line of each Python block is the file name, so you can copy the blocks straight into files.

A few words we will use

When you send a request for a model, Ollama loads that model’s weights into memory, video memory if you have a supported GPU and ordinary memory otherwise. A loaded model then stays in memory for a while so the next request can skip the loading.

A slot is one conversation that a loaded model can work on at the same time as others. A model with one slot answers requests strictly one after another. A model with four slots can generate four answers in the same pass over its weights. A request that arrives when every slot is busy has to wait.

A token is the unit a model reads and writes: a word, a piece of a word or a single digit. Speed is measured in tokens per second. The KV cache is the memory the model uses to remember every token already in a conversation; each slot needs its own, which is why more slots cost more memory. Finally, keep_alive is how long Ollama keeps a model loaded after its last request.

The files at a glance

File What it does Used in
olc.py A tiny Ollama client that returns the HTTP status and the JSON body, even for errors every step
srv.py Starts a throwaway Ollama server with chosen settings and reads back what it logged every step
step01_baseline.py The server’s defaults and the time of one request Step 1
step02_slots.py Several users at once, with 1 to 8 slots Steps 2 and 3
step03_queue.py Which requests get a 503 and which only wait Step 4
step04_keepalive.py What sets the unload timer Step 5
step05_swap.py What switching between models costs Step 6
step06_ignored.py A model that ignores the parallel setting Step 7
gate.py A client-side gate: a bounded waiting room, retries on 503, one explicit keep-alive Step 8
test_gate.py, mutate.py Tests against a fake server, and a check that the tests catch planted bugs Step 8
step07_gate_live.py The gate against the real server Step 8

Step 1: Read the settings the server really uses, and time one request

How Ollama takes its settings

Ollama reads its settings from environment variables when the server starts, so changing one means restarting the server. The ones in this tutorial all begin with OLLAMA_. How you set them depends on how Ollama runs on your machine. In a terminal you can set them for one server and nothing else:

# Linux or macOS shell
OLLAMA_NUM_PARALLEL=4 ollama serve

# Windows PowerShell
$env:OLLAMA_NUM_PARALLEL = "4"; ollama serve

For a permanent setting, Ollama’s FAQ gives one recipe per platform. If Ollama runs as a systemd service, run systemctl edit ollama.service, add a line such as Environment="OLLAMA_NUM_PARALLEL=4" under [Service], then run systemctl daemon-reload and systemctl restart ollama. For the macOS app, call launchctl setenv OLLAMA_NUM_PARALLEL 4 and restart the app. For the Windows app, quit it from the taskbar, add the variable in the “Edit environment variables for your account” dialog, and start Ollama again.

Why the lab reads the server’s own log

Here is a mistake worth making on paper rather than in production. My first run of this lab set a variable called NUM_PARALLEL, without the prefix. Ollama ignores names it does not know, no error appeared, and four “different” configurations produced identical numbers. So the helper below adds the prefix itself, and then does something more important: after the server starts, it reads the settings back from the server’s own log line and refuses to continue if they differ from what you asked for. Trust what the server reports, not what you think you set.

# olc.py
"""A tiny Ollama client for the lab. Standard library only."""
import json
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone


def call(host, path, body=None, timeout=300):
    """Return (http_status, parsed_json). Errors come back as (status, {"error": ..., "_headers": {...}})."""
    data = None if body is None else json.dumps(body).encode()
    request = urllib.request.Request(host + path, data=data, headers={"Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return response.status, json.load(response)
    except urllib.error.HTTPError as err:
        raw = err.read().decode("utf-8", "replace")
        try:
            parsed = json.loads(raw)
        except ValueError:
            parsed = {"error": raw.strip() or str(err.reason)}
        parsed["_headers"] = dict(err.headers)
        return err.code, parsed


def generate(host, model, prompt, num_predict=100, **top):
    """One non-streaming /api/generate call. Returns (status, json, seconds)."""
    options = {"temperature": 0, "seed": 1, "num_predict": num_predict, **top.pop("options", {})}
    body = {"model": model, "prompt": prompt, "stream": False, "options": options, **top}
    started = time.perf_counter()
    status, data = call(host, "/api/generate", body)
    return status, data, time.perf_counter() - started


def ps(host):
    """What /api/ps says is loaded right now."""
    return call(host, "/api/ps")[1].get("models", [])


def unload(host, model):
    call(host, "/api/generate", {"model": model, "keep_alive": 0})


def seconds_left(model_entry):
    """Seconds until the server unloads a model, from the expires_at field of /api/ps."""
    expires = datetime.fromisoformat(model_entry["expires_at"])
    return (expires - datetime.now(timezone.utc)).total_seconds()

The client is the same pattern as in earlier tutorials in this series: call always returns a pair of status and parsed body, so later scripts can inspect a failure without a try block. generate sends one non-streaming request with temperature 0 and a fixed seed so that repeated runs behave alike, and seconds_left will matter in Step 5.

# srv.py
"""Start a throwaway Ollama server with chosen environment variables, then stop it."""
import contextlib
import os
import re
import subprocess
import time

from olc import call

KEYS = ("OLLAMA_NUM_PARALLEL", "OLLAMA_MAX_QUEUE", "OLLAMA_MAX_LOADED_MODELS", "OLLAMA_KEEP_ALIVE", "OLLAMA_CONTEXT_LENGTH")


@contextlib.contextmanager
def server(port, log_name, **env):
    """Run `ollama serve` on 127.0.0.1:<port> with the given OLLAMA_* variables; yield its URL."""
    host = f"127.0.0.1:{port}"
    environment = dict(os.environ, OLLAMA_HOST=host)
    for name in KEYS:
        environment.pop(name, None)
    environment.update({"OLLAMA_" + name: str(value) for name, value in env.items()})
    os.makedirs("out", exist_ok=True)  # the server logs go here, one file per experiment
    log_path = os.path.join("out", f"{log_name}.log")
    with open(log_path, "wb") as log:
        process = subprocess.Popen(["ollama", "serve"], env=environment, stdout=log, stderr=subprocess.STDOUT)
    url = "http://" + host
    try:
        for _ in range(80):
            try:
                call(url, "/api/version", timeout=2)
                break
            except OSError:
                time.sleep(0.25)
        else:
            raise RuntimeError("server did not start")
        logged = config_line(log_name)
        for name, value in env.items():  # trust the server's own log, not the variable we think we set
            if name != "KEEP_ALIVE" and logged[name] != str(value):
                raise RuntimeError(f"server logged {name}={logged[name]}, expected {value}")
        yield url
    finally:
        if os.name == "nt":  # the Windows launcher stub starts a child, so stop the whole tree
            subprocess.run(["taskkill", "/PID", str(process.pid), "/T", "/F"], capture_output=True)
        else:
            process.terminate()
            process.wait(timeout=15)
        time.sleep(0.5)


def config_line(log_name):
    """The server's own report of the settings it started with, from its log."""
    with open(os.path.join("out", f"{log_name}.log"), encoding="utf-8", errors="replace") as handle:
        text = handle.read()
    found = {}
    for name in KEYS:
        match = re.search(name + r":(\S*)", text)
        found[name.replace("OLLAMA_", "")] = match.group(1) if match else "?"
    return found


def log_lines(log_name, pattern):
    """Log lines matching a regex, trimmed of their time stamps."""
    with open(os.path.join("out", f"{log_name}.log"), encoding="utf-8", errors="replace") as handle:
        return [re.sub(r"^time=\S+ ", "", line.strip()) for line in handle if re.search(pattern, line)]

The server function is a context manager. Used in a with block, it starts ollama serve on the port you give it, waits until the server answers on /api/version, checks the logged settings, hands you the server’s address, and stops the server when the block ends. Now the first experiment. It starts a server with every setting left alone, sends the same request twice, and prints what the server reports about the loaded model.

# step01_baseline.py
"""One server with every setting left alone: what does it report, and how fast is one request?"""
from olc import generate, ps
from srv import config_line, log_lines, server

MODEL = "qwen2.5:1.5b"
PROMPT = "Explain in detail how a mechanical wall clock keeps time, starting with the mainspring."

with server(11441, "s01_defaults") as host:
    print("settings the server logged:", config_line("s01_defaults"))
    for attempt in ("cold", "warm"):
        status, data, seconds = generate(host, MODEL, PROMPT, num_predict=100)
        tokens = data["eval_count"]
        rate = tokens / (data["eval_duration"] / 1e9)
        print(f"{attempt}: HTTP {status}  client {seconds:.2f} s  load {data['load_duration'] / 1e9:.2f} s  "
              f"{tokens} tokens at {rate:.0f} tokens/s  total_duration {data['total_duration'] / 1e9:.2f} s")
    for model in ps(host):
        print(f"/api/ps: {model['name']}  context_length={model['context_length']}  size={model['size'] / 2**20:.0f} MiB  "
              f"in VRAM={model['size_vram'] / 2**20:.0f} MiB  expires_at={model['expires_at'][:19]}")
for line in log_lines("s01_defaults", r"parallel|starting runner|runner started|default_num_ctx|OLLAMA_MAX_LOADED"):
    print("log:", line[:230])

Run it with python step01_baseline.py. This is what I got:

settings the server logged: {'NUM_PARALLEL': '1', 'MAX_QUEUE': '512', 'MAX_LOADED_MODELS': '0', 'KEEP_ALIVE': '5m0s', 'CONTEXT_LENGTH': '0'}
cold: HTTP 200  client 3.53 s  load 1.75 s  100 tokens at 59 tokens/s  total_duration 3.52 s
warm: HTTP 200  client 1.88 s  load 0.17 s  100 tokens at 60 tokens/s  total_duration 1.86 s
/api/ps: qwen2.5:1.5b  context_length=4096  size=1112 MiB  in VRAM=1112 MiB  expires_at=2026-11-17T06:27:42
log: level=INFO source=routes.go:1933 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: LLAMA_ARG_FIT: LLAMA_ARG_FIT_
log: level=INFO source=routes.go:2040 msg="vram-based default context" total_vram="4.0 GiB" default_num_ctx=4096

What the output tells you

The first line is the server’s own report of its defaults: one parallel slot (NUM_PARALLEL 1), room for 512 queued requests, an automatic limit on loaded models (MAX_LOADED_MODELS 0 means that the server works the limit out from the hardware), models kept for five minutes (5m0s), and a context length of 0, which means that the server picks a default from the video memory. That matches the FAQ for this version, which says of OLLAMA_NUM_PARALLEL: “The maximum number of parallel requests each model will process at the same time, default 1.” Older versions of the FAQ said something else; the one for version 0.9.0 says: “The default will auto-select either 4 or 1 based on available memory.” For 0.32.9 the FAQ says 1, and so does the source code.

The two requests show the cost of loading. The cold request took 3.53 seconds, of which 1.75 seconds were loading the model. The warm one took 1.88 seconds. Both generated 100 tokens at about 60 tokens per second, so that is this machine’s speed for one conversation. The /api/ps line shows the loaded model: 1,112 MiB, all of it in video memory, with a 4,096-token context window and an expiry time five minutes ahead. Step 5 is about that expiry time.

To verify the step: the first line must show the values you expect (all defaults here), and the warm request must show a small load figure. If a later step prints a settings line that differs from the one you asked for, the helper stops with an error instead of producing misleading numbers.

Step 2: Watch four users share one slot

The experiment

Now we create the situation from the opening paragraph. The script below starts a server with a given number of slots, loads the model with a throwaway request so that loading does not pollute the measurement, and then sends several identical requests at the same instant from separate threads. A threading.Barrier makes every thread wait at a start line and leave together.

For each request it prints three things: when the answer came back, how long the request waited for a free slot, and how fast the model generated. The waiting time is not measured by the client. It is computed from the response itself: total_duration is everything the server did for the request, so subtracting load_duration, prompt_eval_duration and eval_duration leaves the time spent waiting. The same script also runs the other configurations of Step 3, so you will see all of them in one run.

# step02_slots.py
"""Several users at once: one slot versus more, from the client and from the server's own numbers."""
import threading
import time

from olc import generate, ps
from srv import log_lines, server

MODEL = "qwen2.5:1.5b"
PROMPT = "Explain in detail how a mechanical wall clock keeps time, starting with the mainspring."
TOKENS = 100


def burst(host, users):
    """Send `users` identical requests at the same moment. Returns (results, wall seconds)."""
    results = [None] * users
    start_line = threading.Barrier(users + 1)

    def worker(index):
        start_line.wait()
        results[index] = generate(host, MODEL, PROMPT, num_predict=TOKENS)

    threads = [threading.Thread(target=worker, args=(i,)) for i in range(users)]
    for thread in threads:
        thread.start()
    start_line.wait()
    started = time.perf_counter()
    for thread in threads:
        thread.join()
    return results, time.perf_counter() - started


def waited(data):
    """Seconds the request spent waiting for a free slot: the part of total_duration that is not load, prompt or generation."""
    return (data["total_duration"] - data["load_duration"] - data["prompt_eval_duration"] - data["eval_duration"]) / 1e9


summary = []
for slots, users in [(1, 4), (2, 4), (4, 4), (8, 4), (4, 8)]:
    name = f"s02_parallel{slots}_users{users}"
    with server(11450 + slots * 10 + users, name, NUM_PARALLEL=slots) as host:
        generate(host, MODEL, "Say hello.", num_predict=5)  # load the model first so the burst measures only the waiting
        results, wall = burst(host, users)
        model = ps(host)[0]
        print(f"--- OLLAMA_NUM_PARALLEL={slots}, {users} users at once")
        print("user  HTTP  finished after  waited for a slot  tokens/s while generating")
        rows = sorted(results, key=lambda r: r[2])
        for number, (status, data, seconds) in enumerate(rows, 1):
            rate = data["eval_count"] / (data["eval_duration"] / 1e9)
            print(f"{number:>4}  {status}   {seconds:>8.2f} s   {waited(data):>14.2f} s   {rate:>12.0f}")
        tokens = sum(r[1]["eval_count"] for r in results)
        rates = [r[1]["eval_count"] / (r[1]["eval_duration"] / 1e9) for r in results]
        summary.append((slots, users, wall, rows[-1][2], sum(rates) / len(rates), tokens / wall,
                        model["context_length"], model["size"] / 2**20))
        for line in log_lines(name, r"does not currently support"):
            print("log:", line[:200])

print()
print("slots  users  wall s  slowest s  tokens/s per user  tokens/s total  context_length  size MiB")
for slots, users, wall, slowest, per_user, total, context, size in summary:
    print(f"{slots:>5}  {users:>5}  {wall:>6.2f}  {slowest:>9.2f}  {per_user:>17.0f}  {total:>14.0f}  {context:>14}  {size:>8.0f}")

Run it with python step02_slots.py. It takes about a minute, because it starts five servers in turn. First, the default configuration, one slot and four users:

--- OLLAMA_NUM_PARALLEL=1, 4 users at once
user  HTTP  finished after  waited for a slot  tokens/s while generating
   1  200       1.86 s             0.01 s             60
   2  200       3.58 s             1.70 s             59
   3  200       5.29 s             3.41 s             59
   4  200       7.00 s             5.12 s             59

What happened

This is a staircase. The first user finished after 1.86 seconds, which is the time one request takes on this machine. The second finished 1.7 seconds later, the third another 1.7 seconds after that, and the fourth after 7.00 seconds, nearly four times as long as the first. Look at the second column from the right: the second user waited 1.70 seconds for a slot, the third 3.41, the fourth 5.12. That is about the generation time of the requests ahead of them. While each request ran, the model generated at the full 59 to 60 tokens per second, so the speed of one conversation was never the problem. The line was.

The server is not misbehaving. With one slot, the model works on one request at a time, and everyone else waits in order. For one person that is ideal, because they get the fastest possible answer. For four people it means the last one waits nearly four times as long as the first, however much the hardware could do in parallel.

Step 3: Give the model more slots with OLLAMA_NUM_PARALLEL

What a slot buys you

Setting OLLAMA_NUM_PARALLEL to 4 gives each loaded model four slots. The weights, which are most of the memory at small context sizes, are loaded once and shared. What each extra slot adds is a KV cache of its own, because each conversation needs to remember its own tokens. Ollama’s FAQ states the consequence: “Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation.”

You already ran the experiment: the same script just did it for 2, 4 and 8 slots. Here is the four-slot run:

--- OLLAMA_NUM_PARALLEL=4, 4 users at once
user  HTTP  finished after  waited for a slot  tokens/s while generating
   1  200       3.42 s             0.01 s             31
   2  200       3.44 s             0.01 s             32
   3  200       3.44 s             0.02 s             32
   4  200       3.44 s             0.03 s             32

The staircase has vanished. All four users finished at about 3.4 seconds, and nobody waited more than 0.03 seconds for a slot. The price is visible in the last column: each user now generated at 31 or 32 tokens per second instead of 60. The summary table at the end of the output puts all five runs side by side:

slots  users  wall s  slowest s  tokens/s per user  tokens/s total  context_length  size MiB
    1      4    7.00       7.00                 59              57            4096      1112
    2      4    4.26       4.26                 50              94            4096      1290
    4      4    3.44       3.44                 32             116            4096      1514
    8      4    3.42       3.42                 32             117            4096      1962
    4      8    6.63       6.63                 32             121            4096      1514

Reading the table

Three things stand out. First, the slowest user improves a lot: 7.00 seconds with one slot, 4.26 with two, 3.44 with four. Second, each user gets slower while they share: 59 tokens per second alone, 50 with two slots busy, 32 with four. The total, though, grows from 57 to 94 to 116 tokens per second, which fits the idea that the GPU does more useful work when one pass over the weights serves several conversations. Third, the gain stops. Eight slots with four users changed nothing (117 tokens per second), and with eight users on four slots the total rose only to 121. That suggests that four conversations already use up what this small GPU can do, so slots beyond that help only by keeping more requests in flight, not by making the machine faster. A bigger GPU would move that limit.

The last row is the one to remember when you plan for a crowd. With four slots and eight users, the first four finished at 3.4 seconds and the other four at 6.6, after waiting about 3.25 seconds for a slot. Slots set how many requests run at once; everyone beyond that still waits in line.

The memory bill

The size column shows what the extra slots cost: 1,112 MiB with one slot, 1,290 with two, 1,514 with four and 1,962 with eight. A KV cache for one 4,096-token window of this model is 28,672 bytes per token (28 layers, 2 key-value heads, 128 values per head, two bytes each, for both keys and values) times 4,096 tokens, which is 112 MiB. Each extra slot cost a little more than that, between about 120 and 180 MiB on average, which suggests that the server also reserves some scratch space per slot. Notice what did not change: context_length stayed at 4,096 for every configuration. Each user still has the full window. The server simply reserves four windows in total, exactly as the FAQ says.

That is the rule for sizing: memory grows with slots times window. A server with 4 slots and a 32,768-token window needs eight times the cache of one with 4 slots and 4,096 tokens. If you raise OLLAMA_NUM_PARALLEL and your model suddenly runs partly on the CPU, or fails to load, this is the first thing to check. Start from the number of people who really send requests at the same moment, not from the number of people with access, and stop adding slots when the total tokens per second stops rising.

Step 4: Find out when Ollama really answers 503

What the documentation promises

Ollama’s FAQ describes the queue like this: “If too many requests are sent to the server, it will respond with a 503 error indicating the server is overloaded. You can adjust how many requests may be queued by setting OLLAMA_MAX_QUEUE.” The settings list adds: “The maximum number of requests Ollama will queue when busy before rejecting additional requests. The default is 512”. Taken at face value, this says that a server with one slot and a queue limit of 2 should turn away the fourth request in a pile-up. That is a testable claim, so let us test it twice: once against a model that is loaded and busy, and once against a model that is not loaded yet.

The script starts a server with one slot and OLLAMA_MAX_QUEUE=2, which is deliberately tiny so that any rejection shows up quickly. Experiment A sends 8 requests at once to a loaded model. Experiment B unloads the model first and then sends 8 requests at once, three times. Experiment C repeats B with the default queue size of 512. For each burst it prints how many requests got which HTTP status, how long the slowest one took, and how quickly the fastest refusal came back. For one refusal it also prints the body and the names of the response headers.

# step03_queue.py
"""OLLAMA_MAX_QUEUE: which requests can get a 503?"""
import collections
import threading
import time

from olc import generate, ps, unload
from srv import server

MODEL = "qwen2.5:1.5b"
PROMPT = "Explain in detail how a mechanical wall clock keeps time, starting with the mainspring."


def burst(host, users, tokens=40):
    results = [None] * users
    start_line = threading.Barrier(users + 1)

    def worker(index):
        start_line.wait()
        results[index] = generate(host, MODEL, PROMPT, num_predict=tokens)

    threads = [threading.Thread(target=worker, args=(i,)) for i in range(users)]
    for thread in threads:
        thread.start()
    start_line.wait()
    for thread in threads:
        thread.join()
    return results


def tally(results):
    counts = collections.Counter(status for status, _, _ in results)
    text = " + ".join(f"{count} x HTTP {status}" for status, count in sorted(counts.items()))
    slowest = max(seconds for _, _, seconds in results)
    fastest_refusal = min((seconds for status, _, seconds in results if status != 200), default=None)
    refusal = f", fastest refusal {fastest_refusal:.3f} s" if fastest_refusal is not None else ""
    return f"{text}; slowest {slowest:.2f} s{refusal}"


def wait_until_unloaded(host):
    unload(host, MODEL)
    for _ in range(50):
        if not ps(host):
            return
        time.sleep(0.1)


print("A. model loaded and busy, OLLAMA_NUM_PARALLEL=1, OLLAMA_MAX_QUEUE=2, 8 requests at once")
with server(11500, "s03_busy", NUM_PARALLEL=1, MAX_QUEUE=2) as host:
    generate(host, MODEL, "Say hello.", num_predict=5)
    print("  ", tally(burst(host, 8)))

print("B. model NOT loaded, same settings, 8 requests at once, three tries")
with server(11501, "s03_cold", NUM_PARALLEL=1, MAX_QUEUE=2) as host:
    generate(host, MODEL, "Say hello.", num_predict=5)
    for attempt in (1, 2, 3):
        wait_until_unloaded(host)
        results = burst(host, 8)
        print(f"   try {attempt}:", tally(results))
    refused = next((data for status, data, _ in results if status == 503), None)
    if refused:
        print("   refusal body:", {key: value for key, value in refused.items() if key != "_headers"})
        print("   refusal headers:", sorted(key for key in refused["_headers"]))

print("C. model NOT loaded, default OLLAMA_MAX_QUEUE (512), 8 requests at once")
with server(11502, "s03_default", NUM_PARALLEL=1) as host:
    generate(host, MODEL, "Say hello.", num_predict=5)
    wait_until_unloaded(host)
    print("  ", tally(burst(host, 8)))

Run it with python step03_queue.py. It takes about 40 seconds. The output:

A. model loaded and busy, OLLAMA_NUM_PARALLEL=1, OLLAMA_MAX_QUEUE=2, 8 requests at once
   8 x HTTP 200; slowest 5.73 s
B. model NOT loaded, same settings, 8 requests at once, three tries
   try 1: 3 x HTTP 200 + 5 x HTTP 503; slowest 3.96 s, fastest refusal 0.188 s
   try 2: 3 x HTTP 200 + 5 x HTTP 503; slowest 3.94 s, fastest refusal 0.183 s
   try 3: 3 x HTTP 200 + 5 x HTTP 503; slowest 3.94 s, fastest refusal 0.189 s
   refusal body: {'error': 'server busy, please try again.  maximum pending requests exceeded'}
   refusal headers: ['Connection', 'Content-Length', 'Content-Type', 'Date']
C. model NOT loaded, default OLLAMA_MAX_QUEUE (512), 8 requests at once
   8 x HTTP 200; slowest 7.39 s

What happened

In experiment A, all eight requests were answered. Seven of them waited behind the first one, the queue limit was 2, and nobody was refused. The slowest request took 5.73 seconds, which is simply eight short requests in a row. In experiment B, with the model not loaded, the result was identical on all three tries: 3 requests answered and 5 refused with HTTP 503, the refusals coming back after about 0.19 seconds, well before the model had finished loading. In experiment C, the same cold start with the default limit of 512, all eight requests were answered; the slowest took 7.39 seconds, because everyone waited for the model to load and then took their turn.

The refusal body, as the script prints it, is {'error': 'server busy, please try again. maximum pending requests exceeded'}. There are two spaces after the comma in Ollama’s message, so do not match it with a single-space string. The header list shows Connection, Content-Length, Content-Type and Date and nothing else. In particular there is no Retry-After header, so your client has to decide for itself how long to wait.

Why: the queue is a loading queue

The source code of this version explains the pattern. In server/sched.go, the function that hands a request to a model does this:

if runner != nil && !runner.needsReload(c, req) {
	req.useLoadedRunner(runner, s.finishedReqCh)
} else {
	select {
	case s.pendingReqCh <- req:
	default:
		req.errCh <- ErrMaxQueue
	}
}

Read it from the top. If the model is already loaded and the request does not require any change to it, the request goes straight to the loaded model and waits there for a free slot, a path that never touches the queue. Otherwise the request goes into pendingReqCh, a channel whose size is OLLAMA_MAX_QUEUE, so that the scheduler can load or reload the model. If that channel is full, the request is rejected with ErrMaxQueue, which the HTTP layer turns into a 503. A change, in this sense, includes a model that is not loaded, a request whose runner options differ from the loaded ones (a different num_ctx, for example, which you met in the previous tutorial in this series), and a model whose runner no longer answers.

That explains the three experiments. In B the scheduler takes one request off the channel to start loading, two more fit in the channel of size 2, and the other five find it full, so 3 are answered and 5 refused, one more than the queue size, every time. In A and in the busy-model part of every earlier step, the requests never needed the channel.

What this means for you

The queue limit protects the scheduler during loads. It is not a cap on how long the line behind a busy model can grow. If you want a cap on waiting, you must build it yourself, and Step 8 does. Expect 503s in bursts at three moments: right after the server restarts, right after a model was unloaded by its keep_alive timer, and right after a model swap, which are exactly the moments when many clients ask for a model that is not loaded. A client that treats a 503 as a signal to try again shortly will ride through all three. To verify the step on your own server, run the script and check that experiment A shows no 503 and experiment B shows exactly one more success than the queue size set in the script.

Step 5: Control how long a model stays loaded with keep_alive

Why the timer matters

A model that is not in memory has to be loaded before it can answer, and you saw what that costs in Step 1: the first request took 1.75 seconds of loading, a warm one 0.17. A larger model takes far longer. The setting that decides how long a model stays loaded is keep_alive. There are three ways to use it. The environment variable OLLAMA_KEEP_ALIVE sets the server-wide default, which is five minutes. The keep_alive field in a request to /api/generate or /api/chat sets it for that request. And an empty request, with a model name and a keep_alive but no prompt, preloads or unloads a model without generating anything.

The FAQ lists the accepted values: “a duration string (such as ‘10m’ or ‘24h’)”, “a number in seconds (such as 3600)”, “any negative number which will keep the model loaded in memory (e.g. -1 or ‘-1m’)” and “‘0’ which will unload the model immediately after generating a response”. It also says: “The keep_alive API parameter with the /api/generate and /api/chat API endpoints will override the OLLAMA_KEEP_ALIVE setting.” The API reference describes the field as controlling “how long the model will stay loaded into memory following the request (default: 5m)”, which reads as if every request carries its own timer and five minutes is the fallback. Whether that reading holds matters a great deal on a shared server, because it decides who surprises whom, so the experiment below asks it.

The experiment

The script reads the expires_at field of /api/ps, which is the moment the server will unload the model, and prints how many seconds are left after each kind of request. The most useful rows come in pairs: a request that states a value, followed by an ordinary request that states nothing.

# step04_keepalive.py
"""keep_alive: how long does a model stay loaded, and which request decides?"""
import time

from olc import call, generate, ps, seconds_left, unload
from srv import server

MODEL = "qwen2.5:1.5b"


def show(host, label):
    models = ps(host)
    if not models:
        print(f"{label:<54} nothing loaded")
        return
    left = seconds_left(models[0])
    text = "never" if left > 10**9 else f"{left:,.0f} s"
    print(f"{label:<54} unloads in {text:>7}   (expires_at {models[0]['expires_at'][:10]})")


def ask(host, **top):
    return generate(host, MODEL, "Say hello.", num_predict=5, **top)


with server(11510, "s04_default") as host:
    status, data, _ = ask(host)
    print(f"first request after start: load_duration {data['load_duration'] / 1e9:.2f} s")
    show(host, "ordinary request, no keep_alive")
    status, data, _ = ask(host)
    print(f"second request:            load_duration {data['load_duration'] / 1e9:.2f} s")
    ask(host, keep_alive="30s")
    show(host, 'request with "keep_alive": "30s"')
    ask(host)
    show(host, "next ordinary request, no keep_alive")
    call(host, "/api/generate", {"model": MODEL, "keep_alive": -1})
    show(host, 'empty request with "keep_alive": -1 (preload)')
    ask(host)
    show(host, "next ordinary request, no keep_alive")
    ask(host, keep_alive="5m")
    show(host, 'request with "keep_alive": "5m"')
    unload(host, MODEL)
    show(host, 'empty request with "keep_alive": 0')
    ask(host, keep_alive="3s")
    show(host, 'request with "keep_alive": "3s"')
    time.sleep(1)
    show(host, "   1 s later")
    time.sleep(4.5)
    show(host, "   5.5 s later")

with server(11511, "s04_env", KEEP_ALIVE="45s") as host:
    ask(host)
    show(host, "OLLAMA_KEEP_ALIVE=45s, request without keep_alive")
    ask(host, keep_alive="10m")
    show(host, '   same server, request with "keep_alive": "10m"')
    ask(host)
    show(host, "   next ordinary request, no keep_alive")

Run it with python step04_keepalive.py. It takes about 20 seconds. This is what I got:

first request after start: load_duration 1.74 s
ordinary request, no keep_alive                        unloads in   300 s   (expires_at 2026-11-17)
second request:            load_duration 0.14 s
request with "keep_alive": "30s"                       unloads in    30 s   (expires_at 2026-11-17)
next ordinary request, no keep_alive                   unloads in    30 s   (expires_at 2026-11-17)
empty request with "keep_alive": -1 (preload)          unloads in   never   (expires_at 2319-02-27)
next ordinary request, no keep_alive                   unloads in   never   (expires_at 2319-02-27)
request with "keep_alive": "5m"                        unloads in   300 s   (expires_at 2026-11-17)
empty request with "keep_alive": 0                     nothing loaded
request with "keep_alive": "3s"                        unloads in     3 s   (expires_at 2026-11-17)
   1 s later                                           unloads in     2 s   (expires_at 2026-11-17)
   5.5 s later                                         nothing loaded
OLLAMA_KEEP_ALIVE=45s, request without keep_alive      unloads in    45 s   (expires_at 2026-11-17)
   same server, request with "keep_alive": "10m"       unloads in   600 s   (expires_at 2026-11-17)
   next ordinary request, no keep_alive                unloads in   600 s   (expires_at 2026-11-17)

Reading it row by row

The first request after the server started paid a 1.74 second load, the second only 0.14. An ordinary request leaves the model loaded for 300 seconds, the five-minute default. A request with "keep_alive": "30s" sets the remaining time to 30 seconds, as expected. The next row is the surprise: an ordinary request that states nothing did not go back to 300 seconds. It left the rule at 30 seconds. The empty request with -1 pins the model, and Ollama turns a negative value into the largest duration its time type can hold, about 292 years, so the reported expiry lands in the year 2319. The next ordinary request leaves it pinned. Only a request that states a new value, here "5m", changes the rule again. Then 0 unloads the model at once, and "3s" unloads it three seconds after the request, so after 5.5 seconds nothing is loaded and the next caller will pay for a load.

The last three rows repeat the pattern against a server started with OLLAMA_KEEP_ALIVE=45s. An ordinary request gets 45 seconds, a request with "10m" gets 600, and the next ordinary request still gets 600. The environment default did not come back either.

The rule, and why the server behaves this way

The value belongs to the loaded model, not to the request. The most recent request that states a keep_alive sets it, and every later request, with or without one of its own, restarts the countdown with whatever is currently set. You can see this in server/sched.go: when a request reaches an already loaded model, the code reads if pending.sessionDuration != nil { runner.sessionDuration = pending.sessionDuration.Duration }, so a request without the field leaves the stored duration alone. When a model is loaded fresh, it starts from OLLAMA_KEEP_ALIVE unless the loading request states its own value.

On a shared server, that has three consequences. A single client that sends "keep_alive": 0 unloads the model for everyone, and the next user pays the load and may meet the 503 burst from Step 4. A single client that sends -1 pins the model in memory for everyone, even though the server’s default says 45 seconds, until somebody states another value. And a mix of clients that each state a different value leaves the model’s fate to whichever request arrived last.

What to do about it

Choose one policy for the server and make every client agree with it. If a model must always be ready, set OLLAMA_KEEP_ALIVE=-1 in the server’s environment, or preload it once at start-up with curl http://localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "keep_alive": -1}', which is the command the FAQ shows, with this tutorial’s model name. It then holds 1,112 MiB of video memory for as long as the server runs, which is the price of never paying a load. If the GPU is shared with other work, choose a finite value such as "30m" and send that same value from every client you control. Step 8 builds a gate that always sends one explicit value, so none of your own callers can change the rule by accident. To verify on your server, read expires_at from /api/ps after a request: it is the quickest way to see which rule is currently in force.

Step 6: Keep more than one model ready with OLLAMA_MAX_LOADED_MODELS

The second level of concurrency

Slots let one model serve several people. The other level is several models at once. The FAQ describes it: “Ollama supports two levels of concurrent processing. If your system has sufficient available memory (system memory when using CPU inference, or VRAM for GPU inference) then multiple models can be loaded at the same time.” The limit is OLLAMA_MAX_LOADED_MODELS, described as “The maximum number of models that can be loaded concurrently provided they fit in available memory.” The default of 0 means three per GPU, which the source confirms with a constant defaultModelsPerGPU = 3. And the FAQ spells out what happens when memory runs short: “all new requests will be queued until the new model can be loaded. As prior models become idle, one or more will be unloaded to make room for the new model.”

A team that uses a large model for answers and a small one for quick tasks such as classification will alternate between the two all day. The experiment shows what that costs when the two cannot be loaded together and when they can. The script runs the same five requests twice: first on a server limited to one loaded model, then on a server with the default limit. The models alternate: large, small, large, small, small. After each request it prints the load time, the client’s total time and the list of models /api/ps says are loaded. It then unloads both models and sends four requests at once, two for each model.

# step05_swap.py
"""OLLAMA_MAX_LOADED_MODELS: what a model swap costs."""
import collections
import threading
import time

from olc import generate, ps, unload
from srv import server

BIG, SMALL = "qwen2.5:1.5b", "qwen2.5:0.5b"
PROMPT = "Explain in detail how a mechanical wall clock keeps time, starting with the mainspring."


def sequence(host):
    print("request          model load_duration  client time   loaded afterwards")
    for number, model in enumerate([BIG, SMALL, BIG, SMALL, SMALL], 1):
        status, data, seconds = generate(host, model, PROMPT, num_predict=20)
        loaded = ", ".join(sorted(m["name"] for m in ps(host)))
        print(f"{number}  {model:<16} {data['load_duration'] / 1e9:>9.2f} s {seconds:>11.2f} s   {loaded}")


def mixed_burst(host):
    for model in (BIG, SMALL):
        unload(host, model)
    time.sleep(0.5)
    results = []
    start_line = threading.Barrier(5)

    def worker(model):
        start_line.wait()
        results.append(generate(host, model, PROMPT, num_predict=60))

    threads = [threading.Thread(target=worker, args=(m,)) for m in (BIG, SMALL, BIG, SMALL)]
    for thread in threads:
        thread.start()
    start_line.wait()
    started = time.perf_counter()
    for thread in threads:
        thread.join()
    wall = time.perf_counter() - started
    counts = collections.Counter(status for status, _, _ in results)
    print(f"mixed burst (2 x {BIG}, 2 x {SMALL}, cold): wall {wall:.2f} s, statuses {dict(counts)}, "
          f"loaded afterwards: {', '.join(sorted(m['name'] for m in ps(host)))}")


for label, env in [("OLLAMA_MAX_LOADED_MODELS=1", {"MAX_LOADED_MODELS": 1}), ("default (3 per GPU)", {})]:
    print(f"=== {label}")
    with server(11520 + len(env), "s05_" + str(len(env)), **env) as host:
        sequence(host)
        mixed_burst(host)

Run it with python step05_swap.py, which takes about 35 seconds:

=== OLLAMA_MAX_LOADED_MODELS=1
request          model load_duration  client time   loaded afterwards
1  qwen2.5:1.5b          1.75 s        2.18 s   qwen2.5:1.5b
2  qwen2.5:0.5b          2.65 s        2.93 s   qwen2.5:0.5b
3  qwen2.5:1.5b          2.59 s        3.02 s   qwen2.5:1.5b
4  qwen2.5:0.5b          2.31 s        2.56 s   qwen2.5:0.5b
5  qwen2.5:0.5b          0.16 s        0.32 s   qwen2.5:0.5b
mixed burst (2 x qwen2.5:1.5b, 2 x qwen2.5:0.5b, cold): wall 7.21 s, statuses {200: 4}, loaded afterwards: qwen2.5:0.5b
=== default (3 per GPU)
request          model load_duration  client time   loaded afterwards
1  qwen2.5:1.5b          1.73 s        2.17 s   qwen2.5:1.5b
2  qwen2.5:0.5b          1.01 s        1.26 s   qwen2.5:0.5b, qwen2.5:1.5b
3  qwen2.5:1.5b          0.17 s        0.52 s   qwen2.5:0.5b, qwen2.5:1.5b
4  qwen2.5:0.5b          0.17 s        0.33 s   qwen2.5:0.5b, qwen2.5:1.5b
5  qwen2.5:0.5b          0.18 s        0.34 s   qwen2.5:0.5b, qwen2.5:1.5b
mixed burst (2 x qwen2.5:1.5b, 2 x qwen2.5:0.5b, cold): wall 3.39 s, statuses {200: 4}, loaded afterwards: qwen2.5:0.5b, qwen2.5:1.5b

What happened

With a limit of one model, every change of model is a swap. Requests 2, 3 and 4 each paid a load of 2.31 to 2.65 seconds, which is almost the entire time the request took, and only request 5, a repeat of the model in memory, was cheap at 0.16 seconds. With the default limit, only the first use of each model paid a load: 1.73 seconds for the larger model and 1.01 for the smaller one, less than in the swapping run, probably because nothing had to be unloaded first. After that every request, in any order, started in about 0.17 seconds, and /api/ps listed both models.

The mixed burst shows the same thing under load. The four requests took 7.21 seconds with room for one model, because the scheduler had to unload and load while requests waited, and 3.39 seconds with room for both, which fits the two models then generating at the same time. A request for a model that is not loaded also goes through the loading queue from Step 4, so on a server with a small OLLAMA_MAX_QUEUE a swap storm is another place where 503 responses can appear.

Choosing the number

Raising OLLAMA_MAX_LOADED_MODELS helps only if the models really fit in memory together, with each model’s slots and window counted as in Step 3. If they do not fit, the scheduler unloads an idle model whatever the number says, and you are back to swapping. The practical rule is to add up the size values that /api/ps reports for the models you want resident, leave headroom, and set the limit to the number of models in that group. For models that cannot share the GPU, one option is two Ollama servers on different ports, each with a model of its own, behind whatever routes your requests.

Step 7: Spot a parallel setting that the server ignores

The setting that did nothing

One more trap deserves its own step, because it produces no error anywhere you would normally look. Some model families do not support parallel requests, and Ollama quietly runs them with one slot whatever OLLAMA_NUM_PARALLEL says. In server/sched.go of this version, the code is explicit: a comment says “Some architectures are not safe with num_parallel > 1”, followed by a list of families that are forced to one slot: mllama, qwen3vl, qwen3vlmoe, qwen35, qwen35moe, qwen3next, lfm2, lfm2moe, nemotron_h, nemotron_h_moe and nemotron_h_omni. A separate comment says: “Embedding models should always be loaded with parallel=1”.

The script below starts a server with OLLAMA_NUM_PARALLEL=4 and loads qwen3.5:4b, a model from the qwen35 family. It times one user alone, then four users at once, and then prints the window that /api/ps reports and any server log line about parallel requests. The requests set think to false and ask for 30 tokens so that the model answers quickly instead of reasoning at length.

# step06_ignored.py
"""A setting the server accepts and then ignores for some models."""
import threading
import time

from olc import generate, ps
from srv import log_lines, server

MODEL = "qwen3.5:4b"
PROMPT = "Name four kinds of clock and say in one sentence how each one keeps time."

with server(11530, "s06_ignored", NUM_PARALLEL=4) as host:
    generate(host, MODEL, "Say hello.", num_predict=5, think=False)
    status, data, alone = generate(host, MODEL, PROMPT, num_predict=30, think=False)
    print(f"one user alone: {alone:.2f} s ({data['eval_count']} tokens)")
    results = [None] * 4
    start_line = threading.Barrier(5)

    def worker(index):
        start_line.wait()
        results[index] = generate(host, MODEL, PROMPT, num_predict=30, think=False)

    threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
    for thread in threads:
        thread.start()
    start_line.wait()
    started = time.perf_counter()
    for thread in threads:
        thread.join()
    wall = time.perf_counter() - started
    finish = sorted(round(r[2], 2) for r in results)
    print(f"four users at once with OLLAMA_NUM_PARALLEL=4: wall {wall:.2f} s, finished after {finish} s")
    print("context_length reported:", [m["context_length"] for m in ps(host)])
for line in log_lines("s06_ignored", r"does not currently support"):
    print("log:", line[:220])

Run it with python step06_ignored.py. The first load of this larger model takes a while, so allow a minute:

one user alone: 2.40 s (30 tokens)
four users at once with OLLAMA_NUM_PARALLEL=4: wall 7.57 s, finished after [2.17, 3.97, 5.77, 7.57] s
context_length reported: [4096]
log: level=WARN source=sched.go:509 msg="model architecture does not currently support parallel requests" architecture=qwen35

What happened

The settings check in srv.py confirmed that the server logged NUM_PARALLEL 4, yet the four users finished after 2.17, 3.97, 5.77 and 7.57 seconds, a staircase with a step of about 1.8 seconds. That is the one-slot pattern from Step 2. The window that /api/ps reports stayed at 4,096, as it did in Step 3 for every slot count, so that field cannot tell you how many slots a model really has, and nothing else in the HTTP responses says so either. The only trace is a warning in the server’s log: msg="model architecture does not currently support parallel requests" architecture=qwen35.

How to catch it on your server

After changing OLLAMA_NUM_PARALLEL, always run the Step 2 script with the model you really serve and look at whether the staircase went away. You can also search the log. Ollama’s troubleshooting page gives the places to look: on a Linux systemd install, journalctl -u ollama --no-pager --follow --pager-end; on macOS, ~/.ollama/logs/server.log; on Windows, the server.log file in %LOCALAPPDATA%\Ollama. Search for “does not currently support parallel requests”. The list of families belongs to version 0.32.9 and will change as engines improve, so treat it as an example of the problem rather than as a list to memorize.

Step 8: Protect your own application with a small client-side gate

Three jobs for the gate

Steps 2 to 7 give you three jobs for a small piece of code that sits between your application and Ollama. First, cap the number of requests in flight at the number of slots, so that a burst never reaches the loading queue all at once (Step 4), and cap the number allowed to wait, so that when demand truly exceeds capacity your application can tell its callers at once that it is busy and to try later, instead of letting them hang behind a line that has no limit. Second, retry the 503 that can still happen at a restart or a model swap, with a growing, randomized pause and a deadline, and fail at once on any other error. Third, send one explicit keep_alive on every call, so that none of your own callers can change the rule by accident (Step 5).

The general pattern is the bulkhead from an earlier tutorial in this series, How to Build a Bulkhead in Python to Stop a Slow Dependency From Starving the Rest, and the retry rules come from the retry tutorial with exponential backoff and jitter. The gate below is a small, Ollama-specific version of both.

The code

# gate.py
"""A client-side gate for a shared Ollama server: a bounded waiting room, 503 retries with jitter, explicit keep_alive."""
import random
import threading
import time

from olc import call


class GateError(Exception):
    """Base class for everything the gate raises."""


class Rejected(GateError):
    """No slot was free soon enough. Answer your own caller with a 429, or ask it to try later."""


class ServerBusy(GateError):
    """Ollama kept answering 503 until the deadline ran out."""


class OllamaError(GateError):
    """Ollama answered with an error that retrying will not fix."""

    def __init__(self, status, message):
        super().__init__(f"HTTP {status}: {message}")
        self.status = status


class Gate:
    def __init__(self, host, slots, max_waiting, deadline=30.0, keep_alive="10m",
                 base=0.25, cap=4.0, sleep=time.sleep, clock=time.monotonic, rng=random.random):
        self.host = host
        self.max_waiting = max_waiting
        self.deadline = deadline
        self.keep_alive = keep_alive
        self.base = base
        self.cap = cap
        self.sleep = sleep
        self.clock = clock
        self.rng = rng
        self.slots = threading.BoundedSemaphore(slots)
        self.waiting = 0
        self.retried = 0
        self.lock = threading.Lock()

    def generate(self, model, prompt, **fields):
        """Run one /api/generate call under the gate and return Ollama's JSON answer."""
        started = self.clock()
        if not self.slots.acquire(blocking=False):
            with self.lock:
                if self.waiting >= self.max_waiting:
                    raise Rejected(f"{self.waiting} callers are already waiting for a slot")
                self.waiting += 1
            try:
                got_slot = self.slots.acquire(timeout=self.deadline)
            finally:
                with self.lock:
                    self.waiting -= 1
            if not got_slot:
                raise Rejected(f"no slot became free within {self.deadline} s")
        try:
            return self._call(model, prompt, fields, started)
        finally:
            self.slots.release()

    def _call(self, model, prompt, fields, started):
        body = {"model": model, "prompt": prompt, "stream": False, **fields}
        body.setdefault("keep_alive", self.keep_alive)
        attempt = 0
        while True:
            left = self.deadline - (self.clock() - started)
            status, data = call(self.host, "/api/generate", body, timeout=max(1.0, left))
            if status == 200:
                return data
            if status != 503:
                raise OllamaError(status, data.get("error", ""))
            delay = self.rng() * min(self.cap, self.base * 2 ** attempt)
            if self.clock() - started + delay > self.deadline:
                raise ServerBusy(f"still busy after {attempt + 1} tries")
            with self.lock:
                self.retried += 1
            self.sleep(delay)
            attempt += 1

Read generate first. It tries to take a slot without waiting. If none is free, it checks the waiting room: when max_waiting callers are already waiting, it raises Rejected immediately, and your web handler can turn that into an HTTP 429 response. Otherwise the caller counts itself into the room, waits for a slot for at most deadline seconds, and counts itself out again, whatever happens. Whoever holds a slot always gives it back in a finally block.

Now _call. It builds the request body and adds keep_alive only if the caller did not set one, so the gate’s value is the default and a caller can still ask for an unload on purpose. A 200 returns the answer. Any status other than 503 raises OllamaError at once, because retrying a bad request only repeats the mistake. A 503 sleeps for a random time between zero and the smaller of cap and base doubled once per attempt, which spreads the retries of many clients over time instead of sending them back in a block, and gives up with ServerBusy when the next sleep would pass the deadline. The clock, the sleep function and the random function are parameters so that tests can control them.

Tests against a fake server

The tests start a tiny fake Ollama in a thread. It can refuse the first few calls with 503, answer slowly, or always fail, and it records every request body and how many requests ran at the same time. A fake clock that only moves when the gate sleeps makes the backoff tests instant.

# test_gate.py
"""Tests for gate.py against a fake Ollama server, so they need no model and run in seconds."""
import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import pytest

from gate import Gate, OllamaError, Rejected, ServerBusy


class FakeOllama:
    """Answers /api/generate. Can refuse the first `busy_first` calls with 503, or always fail with `fail_status`."""

    def __init__(self, busy_first=0, delay=0.0, fail_status=None):
        self.busy_first = busy_first
        self.delay = delay
        self.fail_status = fail_status
        self.bodies = []
        self.in_flight = 0
        self.max_in_flight = 0
        self.lock = threading.Lock()
        fake = self

        class Handler(BaseHTTPRequestHandler):
            def do_POST(self):
                body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
                with fake.lock:
                    fake.bodies.append(body)
                    fake.in_flight += 1
                    fake.max_in_flight = max(fake.max_in_flight, fake.in_flight)
                    refuse = fake.busy_first > 0
                    fake.busy_first -= 1 if refuse else 0
                time.sleep(fake.delay)
                with fake.lock:
                    fake.in_flight -= 1
                if refuse:
                    status, payload = 503, {"error": "server busy, please try again.  maximum pending requests exceeded"}
                elif fake.fail_status:
                    status, payload = fake.fail_status, {"error": "nope"}
                else:
                    status, payload = 200, {"response": "ok", "done": True}
                raw = json.dumps(payload).encode()
                self.send_response(status)
                self.send_header("Content-Length", str(len(raw)))
                self.end_headers()
                self.wfile.write(raw)

            def log_message(self, *args):
                pass

        self.httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
        self.url = f"http://127.0.0.1:{self.httpd.server_address[1]}"
        threading.Thread(target=self.httpd.serve_forever, daemon=True).start()

    def close(self):
        self.httpd.shutdown()
        self.httpd.server_close()


@pytest.fixture
def make_fake():
    made = []

    def make(**kwargs):
        fake = FakeOllama(**kwargs)
        made.append(fake)
        return fake

    yield make
    for fake in made:
        fake.close()


class Clock:
    """A clock that only moves when the gate sleeps, so backoff tests take no real time."""

    def __init__(self):
        self.now = 0.0
        self.sleeps = []

    def time(self):
        return self.now

    def sleep(self, seconds):
        self.sleeps.append(seconds)
        self.now += seconds


def test_every_call_carries_keep_alive(make_fake):
    fake = make_fake()
    gate = Gate(fake.url, slots=1, max_waiting=1, keep_alive="10m")
    assert gate.generate("m", "hi")["response"] == "ok"
    gate.generate("m", "hi", keep_alive=0)
    assert [body["keep_alive"] for body in fake.bodies] == ["10m", 0]


def test_never_more_in_flight_than_slots(make_fake):
    fake = make_fake(delay=0.05)
    gate = Gate(fake.url, slots=2, max_waiting=8)
    threads = [threading.Thread(target=gate.generate, args=("m", "hi")) for _ in range(8)]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
    assert len(fake.bodies) == 8
    assert fake.max_in_flight == 2


def test_503_is_retried_until_it_works(make_fake):
    fake = make_fake(busy_first=3)
    clock = Clock()
    gate = Gate(fake.url, slots=1, max_waiting=1, sleep=clock.sleep, clock=clock.time)
    assert gate.generate("m", "hi")["response"] == "ok"
    assert len(fake.bodies) == 4
    assert len(clock.sleeps) == 3
    assert gate.retried == 3


def test_backoff_doubles_up_to_the_cap(make_fake):
    fake = make_fake(busy_first=6)
    clock = Clock()
    gate = Gate(fake.url, slots=1, max_waiting=1, deadline=60, base=0.25, cap=4.0,
                sleep=clock.sleep, clock=clock.time, rng=lambda: 1.0)
    gate.generate("m", "hi")
    assert clock.sleeps == [0.25, 0.5, 1.0, 2.0, 4.0, 4.0]


def test_backoff_is_jittered(make_fake):
    fake = make_fake(busy_first=3)
    clock = Clock()
    gate = Gate(fake.url, slots=1, max_waiting=1, sleep=clock.sleep, clock=clock.time, rng=lambda: 0.5)
    gate.generate("m", "hi")
    assert clock.sleeps == [0.125, 0.25, 0.5]


def test_gives_up_at_the_deadline(make_fake):
    fake = make_fake(busy_first=10**6)
    clock = Clock()
    gate = Gate(fake.url, slots=1, max_waiting=1, deadline=5.0, sleep=clock.sleep, clock=clock.time, rng=lambda: 1.0)
    with pytest.raises(ServerBusy):
        gate.generate("m", "hi")
    assert sum(clock.sleeps) <= 5.0
    assert len(fake.bodies) == 5


def test_other_errors_are_not_retried(make_fake):
    fake = make_fake(fail_status=400)
    gate = Gate(fake.url, slots=1, max_waiting=1)
    with pytest.raises(OllamaError) as caught:
        gate.generate("m", "hi")
    assert caught.value.status == 400
    assert len(fake.bodies) == 1


def test_a_failed_call_gives_its_slot_back(make_fake):
    fake = make_fake(fail_status=500)
    gate = Gate(fake.url, slots=1, max_waiting=0, deadline=2.0)
    for _ in range(3):
        with pytest.raises(OllamaError):
            gate.generate("m", "hi")
    assert len(fake.bodies) == 3


def test_a_full_waiting_room_rejects_at_once(make_fake):
    fake = make_fake(delay=0.6)
    gate = Gate(fake.url, slots=1, max_waiting=1, deadline=5.0)
    first = threading.Thread(target=gate.generate, args=("m", "hi"))
    first.start()
    while fake.in_flight < 1:
        time.sleep(0.01)
    second = threading.Thread(target=gate.generate, args=("m", "hi"))
    second.start()
    while gate.waiting < 1:
        time.sleep(0.01)
    started = time.perf_counter()
    with pytest.raises(Rejected):
        gate.generate("m", "hi")
    assert time.perf_counter() - started < 0.2
    first.join()
    second.join()
    assert gate.waiting == 0
    assert len(fake.bodies) == 2


def test_no_waiting_room_means_no_queue(make_fake):
    fake = make_fake(delay=0.4)
    gate = Gate(fake.url, slots=1, max_waiting=0)
    first = threading.Thread(target=gate.generate, args=("m", "hi"))
    first.start()
    while fake.in_flight < 1:
        time.sleep(0.01)
    with pytest.raises(Rejected):
        gate.generate("m", "hi")
    first.join()
    assert gate.generate("m", "hi")["response"] == "ok"


def test_waiting_too_long_for_a_slot_is_a_rejection(make_fake):
    fake = make_fake(delay=1.0)
    gate = Gate(fake.url, slots=1, max_waiting=1, deadline=0.3)
    first = threading.Thread(target=gate.generate, args=("m", "hi"))
    first.start()
    while fake.in_flight < 1:
        time.sleep(0.01)
    with pytest.raises(Rejected):
        gate.generate("m", "hi")
    first.join()
    assert gate.waiting == 0

Run them with python -m pytest -q test_gate.py:

...........                                                                                                                                                        [100%]
11 passed in 8.00s

Each test locks in one decision from this tutorial. test_every_call_carries_keep_alive checks Step 5, and also that a caller’s own value wins. test_never_more_in_flight_than_slots sends eight callers through two slots and checks that the fake never saw more than two at once. The backoff tests pin the sequence 0.25, 0.5, 1, 2, 4, 4 seconds with the random factor fixed at 1, and the sequence 0.125, 0.25, 0.5 with it fixed at 0.5. The waiting-room tests check that a caller is turned away in well under a fifth of a second when the room is full, that a room of size zero means no queue at all, and that the waiting count returns to zero afterwards.

Do the tests really guard anything?

Passing tests prove little if they would also pass with the bugs you fear. So the next script plants one bug at a time in gate.py, runs the tests, and records whether they noticed. It restores the original file when it finishes.

# mutate.py
"""Plant one bug at a time in gate.py and check that the tests notice."""
import os
import pathlib
import shutil
import subprocess
import sys

SOURCE = pathlib.Path("gate.py")
MUTANTS = [
    ("never send keep_alive", 'body.setdefault("keep_alive", self.keep_alive)', "pass"),
    ("override the caller's keep_alive", 'body.setdefault("keep_alive", self.keep_alive)', 'body["keep_alive"] = self.keep_alive'),
    ("one slot too many", "threading.BoundedSemaphore(slots)", "threading.BoundedSemaphore(slots + 1)"),
    ("retry every error", "if status != 503:", "if False:"),
    ("never retry a 503", "if status != 503:", "if True:"),
    ("no jitter", "delay = self.rng() * min(", "delay = 1.0 * min("),
    ("no cap on the backoff", "min(self.cap, self.base * 2 ** attempt)", "self.base * 2 ** attempt"),
    ("no doubling", "self.base * 2 ** attempt)", "self.base * 1 ** attempt)"),
    ("ignore the deadline", "if self.clock() - started + delay > self.deadline:", "if False:"),
    ("waiting room one too big", "if self.waiting >= self.max_waiting:", "if self.waiting > self.max_waiting:"),
    ("forget to count callers out", "self.waiting -= 1", "pass"),
    ("wait for a slot forever", "self.slots.acquire(timeout=self.deadline)", "self.slots.acquire()"),
    ("keep the slot after an error",
     "        try:\n            return self._call(model, prompt, fields, started)\n        finally:\n            self.slots.release()",
     "        result = self._call(model, prompt, fields, started)\n        self.slots.release()\n        return result"),
]


def run_tests():
    shutil.rmtree("__pycache__", ignore_errors=True)
    env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", COLUMNS="170")
    try:
        result = subprocess.run(
            [sys.executable, "-m", "pytest", "-q", "-x", "-p", "no:cacheprovider", "test_gate.py"],
            capture_output=True, text=True, env=env, timeout=120,
        )
    except subprocess.TimeoutExpired:
        return 1, "hung"
    failed = [line for line in result.stdout.splitlines() if line.startswith("FAILED")]
    return result.returncode, failed[0].split(" - ")[0].replace("FAILED test_gate.py::", "") if failed else ""


original = SOURCE.read_text()
code, _ = run_tests()
print(f"clean source: tests exit code {code}")
caught = 0
try:
    for name, old, new in MUTANTS:
        assert original.count(old) == 1, name
        SOURCE.write_text(original.replace(old, new))
        code, first = run_tests()
        caught += code != 0
        print(f"{name:<34} {'caught by ' + first if code else 'SURVIVED'}")
finally:
    SOURCE.write_text(original)
print(f"{caught} of {len(MUTANTS)} planted bugs caught")

Run it with python mutate.py, which takes about a minute and a half:

clean source: tests exit code 0
never send keep_alive              caught by test_every_call_carries_keep_alive
override the caller's keep_alive   caught by test_every_call_carries_keep_alive
one slot too many                  caught by test_never_more_in_flight_than_slots
retry every error                  caught by test_other_errors_are_not_retried
never retry a 503                  caught by test_503_is_retried_until_it_works
no jitter                          caught by test_backoff_is_jittered
no cap on the backoff              caught by test_backoff_doubles_up_to_the_cap
no doubling                        caught by test_backoff_doubles_up_to_the_cap
ignore the deadline                caught by test_gives_up_at_the_deadline
waiting room one too big           caught by test_a_full_waiting_room_rejects_at_once
forget to count callers out        caught by test_a_full_waiting_room_rejects_at_once
wait for a slot forever            caught by test_waiting_too_long_for_a_slot_is_a_rejection
keep the slot after an error       caught by test_a_failed_call_gives_its_slot_back
13 of 13 planted bugs caught

All 13 planted bugs were caught, and in each case the first test to fail is the one written for that decision. If one had survived, the fix would be a new test, not a new bug report.

The gate against the real server

Finally, the live check. The server has two slots and a queue limit of 2, which is the tiny limit from Step 4. Twelve callers arrive at the same moment, and the model is not loaded. The script runs four rounds, unloading the model before each: raw clients with no gate, a gate with two slots and room for 12 waiting callers, a gate with room for only 4, and a gate that allows 12 requests at once, which is more than this server can take.

# step07_gate_live.py
"""The gate against a real server: 12 callers arrive together, the model is cold, and OLLAMA_MAX_QUEUE is small."""
import collections
import threading
import time

from gate import Gate, GateError, Rejected
from olc import generate, ps, unload
from srv import server

MODEL = "qwen2.5:1.5b"
PROMPT = "Explain in detail how a mechanical wall clock keeps time, starting with the mainspring."
OPTIONS = {"temperature": 0, "seed": 1, "num_predict": 40}
CALLERS = 12


def make_cold(host):
    unload(host, MODEL)
    for _ in range(50):
        if not ps(host):
            return
        time.sleep(0.1)


def burst(work):
    outcomes = [None] * CALLERS
    start_line = threading.Barrier(CALLERS + 1)

    def worker(index):
        start_line.wait()
        outcomes[index] = work()

    threads = [threading.Thread(target=worker, args=(i,)) for i in range(CALLERS)]
    for thread in threads:
        thread.start()
    start_line.wait()
    started = time.perf_counter()
    for thread in threads:
        thread.join()
    return collections.Counter(outcomes), time.perf_counter() - started


def raw_call(host):
    status, _, _ = generate(host, MODEL, PROMPT, options=OPTIONS)
    return f"HTTP {status}"


def gated_call(gate):
    try:
        gate.generate(MODEL, PROMPT, options=OPTIONS)
        return "answered"
    except Rejected:
        return "rejected by the gate"
    except GateError as err:
        return type(err).__name__


def show(label, counts, wall, extra=""):
    text = ", ".join(f"{count} x {what}" for what, count in sorted(counts.items()))
    print(f"{label:<44} {text}; wall {wall:.1f} s{extra}")


with server(11540, "s07_gate", NUM_PARALLEL=2, MAX_QUEUE=2) as host:
    generate(host, MODEL, "Say hello.", num_predict=5)
    make_cold(host)
    counts, wall = burst(lambda: raw_call(host))
    show("raw clients, cold model", counts, wall)

    make_cold(host)
    gate = Gate(host, slots=2, max_waiting=CALLERS, deadline=60)
    counts, wall = burst(lambda: gated_call(gate))
    show("gate: 2 slots, room for 12 waiting", counts, wall, f", {gate.retried} retries")

    make_cold(host)
    gate = Gate(host, slots=2, max_waiting=4, deadline=60)
    counts, wall = burst(lambda: gated_call(gate))
    show("gate: 2 slots, room for 4 waiting", counts, wall, f", {gate.retried} retries")

    make_cold(host)
    gate = Gate(host, slots=CALLERS, max_waiting=0, deadline=60)
    counts, wall = burst(lambda: gated_call(gate))
    show("gate: 12 slots (more than the server takes)", counts, wall, f", {gate.retried} retries")

Run it with python step07_gate_live.py:

raw clients, cold model                      3 x HTTP 200, 9 x HTTP 503; wall 4.0 s
gate: 2 slots, room for 12 waiting           12 x answered; wall 7.6 s, 0 retries
gate: 2 slots, room for 4 waiting            6 x answered, 6 x rejected by the gate; wall 4.6 s, 0 retries
gate: 12 slots (more than the server takes)  12 x answered; wall 6.8 s, 37 retries

Reading the four rounds

Without a gate, three of the twelve callers were answered and nine got a 503, exactly the Step 4 pattern. With two slots and room for all twelve, every caller was answered with no retries at all, because the gate never let more than two requests reach the scheduler, which the queue of 2 can always take. The price is time: the last callers waited their turn, so the burst took longer than the raw run. With room for only four waiting callers, six were answered and six were turned away immediately by the gate; those six are the ones you would answer with a 429. The last round shows the retry path doing its job. The gate allowed all twelve through at once, the server refused some of them with 503, and the gate retried each one after a short random pause until it was answered, so the callers saw no error. The retry count at the end of that line is how many 503 responses the gate absorbed: 37 in my run.

One limit to remember: the gate counts only the requests that pass through it, in one process. If you run several worker processes, divide the slots among them so that the total stays at the server’s slot count, or put the gate in one small service that all workers call.

Check that everything works end to end

When you have run all the steps, the picture should be consistent. Step 1 printed the settings you meant to test. Step 2 showed the staircase for one slot and Step 3 showed it shrinking as slots were added, with the total tokens per second rising and then flattening. Step 4 showed no 503 for a busy model and a fixed number of 503 for a cold one. Step 5 showed a keep-alive value surviving ordinary requests. Step 6 showed swaps costing a load each time unless both models fit. Step 7 showed a staircase that no slot setting could remove. And the tests and the live gate run agreed with all of it. If any step disagrees on your machine, trust the measurement over this article: the shapes should match, the numbers will not.

Common mistakes

The first is changing a variable and forgetting that the server reads it only at start-up, so a restart is part of every change. The second is a misspelled variable, which Ollama ignores without a word; read the settings back from the log, as srv.py does. The third is raising OLLAMA_NUM_PARALLEL without counting the memory: slots times window decides the size, and if size_vram in /api/ps is smaller than size, part of the model has moved to the CPU and every user gets slower. The fourth is believing that OLLAMA_MAX_QUEUE limits the wait behind a busy model; it limits the loading queue only. The fifth is letting different clients state different keep_alive values, so that the last one to speak decides for everybody. The sixth is planning from this lab’s identical 100-token requests: real prompts differ in length, a long prompt holds a slot longer, so replay a sample of your own traffic through the Step 2 script before you settle on a number.

Where to go next

The setting that decides how long a prompt can be is the context window, covered in How to Stop Ollama From Silently Cutting Your Prompt, and it multiplies with the slot count as you saw in Step 3. For a careful way to time a local model, read How to Correctly Benchmark LLM Inference Latency With Python and Ollama. For the failure-handling side, the tutorials on circuit breakers, on HTTP timeouts and deadlines and on idempotency keys pair well with the gate. A good exercise is to extend gate.py to read the /api/ps data and refuse to start when the server’s window and slot count do not match what your application assumes.

Sources

The behavior described here comes from experiments on Ollama 0.32.9 and from reading that version’s code and documentation: the FAQ, the troubleshooting page, server/sched.go and envconfig/config.go. Defaults and internals change between releases, so rerun the scripts after you upgrade.

Tags:

ConcurrencyLLM InferenceLLM ServingLocal LLMOllamaPerformance Engineering

Share

Oversized rusty pruning shears sculpture standing above a vineyard path in Langenlois, Austria, under a blue sky
Previous Post

Microsoft’s TokenEconomizer Turns Coding-Agent Cost Into a Learned Policy, but Its 18.4% to 30.9% Was Measured on the Paper’s Controller, Not the Plugin’s Defaults: Micro-Skills Ship Off, the Plugin Never Stops a Turn, and No Pass@1 Gain Clears a Three-Seed Test

Empty clear plastic Petri dish on a white surface
Next Post

Optimizely Says a Signup-Flow Flaw Let an Attacker Plant Malware in Projects Its CDN Served to 178 Organizations, but Not How the Flaw Reached Other Customers, and the CSP Style Its Docs Recommend Does Not Stop the Loader

No Comment! Be the first one.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Latest
22 Nov
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
22 Nov
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
Trending
November 22, 2026
How to Retire an API Endpoint in FastAPI: Send Deprecation and Sunset Headers, Find the Clients Still Calling It, Schedule Brownouts, and Block an Early Removal With oasdiff
November 22, 2026
MIT Technology Review’s “Most Misunderstood Graph” Update Reports METR’s Judgment Findings Mostly Fairly, but It Never Says Each Point Is a 50% Success Rate or Gives the New 3.6-Month Doubling Time and 2.1 to 4.9 Month Range, and It Credits OpenAI With a Comparison METR Built
November 22, 2026
Rackspace Says Leaked Credentials Let an Intruder Act as Legitimate Users in Some Spot Customer Organizations and Reach Their Kubernetes Secrets, a Platform9 Co-Founder Names a Breached Observability System as the Root Cause, and the Status Page Says Only “System Maintenance Operations”
November 21, 2026
How to Write YARA Rules With YARA-X to Hunt Malicious Python .pth Startup Files and Measure False Positives Before You Trust Them
November 21, 2026
AWS’s “Found a Zero-Day on Their Own” Post Never Names the Product; the Advisory Behind Its Link Shows a Server That Ships With No Login, a README Now Saying Keep It Off the Internet, and a Fix Merged October 19 That PyPI and Docker Hub Still Do Not Carry
November 21, 2026
Microsoft Tells Customers Unauthorized Parties Read Admin Email Addresses and Tenant Metadata Through an Entra Connect Health Flaw Introduced July 23, but Gives No Tenant Count or Start Date, and Its Own Docs Send Tenant Security Notices to the Same Kinds of Contacts

Related Posts

A laptop wrapped in a chain and padlock, illustrating least-privilege controls for AI agents.
Learning Hub

How to Secure Tool-Using AI Agents Before They Touch Production

June 8, 2026
Colorful sticky notes arranged on an office wall, symbolizing governance checklists and planning.
Learning Hub

AI Governance for Agentic Apps: A Practical Checklist for Builders

June 8, 2026
A technician connects green fiber optic cables at a data center, representing a private production inference endpoint.
Learning Hub

How to Deploy a Fine-Tuned LLM Behind a Private Production Inference Endpoint

June 8, 2026
Narrow aisle behind black supercomputer racks in a data center
Learning Hub

Kubernetes SELinux Volume Labeling: What Cluster Operators Should Audit Before v1.37

June 8, 2026
SXZ.io SXZ.io
  • [email protected]

Categories

Articles
Learning Hub
News

All Rights Reserved by SXZ.io ©2026