How to Correctly Benchmark LLM Inference Latency With Python and Ollama

If you have ever asked “is this LLM fast enough for production,” you have probably run one request, glanced at how long it took, and moved on. That single number is close to meaningless. LLM inference latency varies request to request, the very first request after a model loads is always slow, and the metric that matters most to a real user (how long until they see the first word of an answer) is not the same metric that matters for total system throughput. This tutorial teaches you how to benchmark LLM inference the way engineers who build serving infrastructure actually do it: with warm-up runs, percentiles instead of averages, and an understanding of exactly what each number is measuring.

By the end, you will have built a real benchmark harness in Python, run it against a local model with Ollama, and personally reproduced five genuine measurement traps that quietly corrupt naive benchmarks, including one that inflated a “time to first token” number by nearly 80x. Everything in this post was run on a real local machine; none of the numbers are invented.

What Benchmarking LLM Inference Actually Means

“How fast is this model” is not one question, it is at least three, and they can move in opposite directions from each other:

• Time to First Token (TTFT): how long a user waits after sending a request before anything appears on screen. This is what makes a chat interface feel responsive or sluggish.
• Time Per Output Token (TPOT), also called Inter-Token Latency (ITL): once the answer starts streaming, how long between each subsequent token. This is what makes the text feel like it is “typing” smoothly or crawling.
• Throughput: how many tokens per second the system produces in total, often measured across many simultaneous requests. This is what determines how many users a server can serve at once.

These definitions are not something this post invented. vLLM, one of the most widely used LLM serving engines, defines TTFT in its own benchmarking documentation as “the time from sending a request to receiving its first streamed output,” and TPOT as the end-to-end latency minus TTFT, divided by the number of output tokens minus one, aggregated across requests. vLLM’s benchmark tool reports these as percentiles (P50, P90, P99), not averages, which is the first hint that averages are not considered good enough by the people who build this infrastructure for a living. You can read the exact wording in vLLM’s Benchmark CLI documentation.

Why a Single Measurement Cannot Answer the Question

A single timed request tells you the latency of that one request, under that one set of conditions, at that one moment. It cannot tell you what a typical user will experience, because “typical” is a statement about a distribution, not a single point. It cannot tell you what happens under load, because a lone request never has to wait behind other requests. And if you happen to measure the very first request after the model was loaded, it cannot even tell you what a typical single request looks like, because that first request pays a cost none of the others will pay. The rest of this tutorial builds up, step by step, the measurement discipline needed to avoid each of these traps, using real captured numbers at every step.

Prerequisites

• Windows, macOS, or Linux with Python 3.10 or newer (this tutorial was built and tested on Python 3.13.14).
• Ollama installed and running locally (this tutorial used Ollama 0.32.9), with at least one model pulled. This tutorial uses qwen2.5:1.5b as the primary benchmark model and qwen3.5:4b for one specific demonstration involving reasoning models. Pull them with ollama pull qwen2.5:1.5b and ollama pull qwen3.5:4b if you do not already have them.
• The requests library: pip install requests (this tutorial used version 2.34.2).
• No GPU is required. Everything here runs fine on CPU, just slower in absolute terms; the methodology and the traps are identical either way.
• Basic comfort running Python scripts from a terminal.

Everything you build here works against any server that speaks Ollama’s HTTP API, and the underlying concepts (TTFT, percentiles, warm-up, concurrency) apply identically to vLLM, TGI, llama.cpp’s server, or a hosted API. Only the request and response JSON shape would need to change.

Step 1: Confirm Your Inference Server Is Reachable

Before benchmarking anything, verify the server actually responds. This sounds trivial, but it matters: if your very first “benchmark” call also has to establish that the server is up, you will bake connection setup time into your very first measurement without realizing it.

curl http://127.0.0.1:11434/api/version

Expected output:

{"version":"0.32.9"}

Notice this uses 127.0.0.1, not localhost. Keep that in mind; it becomes important, and surprising, in Step 3.

Step 2: The Naive Benchmark (and Why It Lies to You)

Here is the benchmark almost everyone writes first: send one request, time it, done.

import time
import requests

OLLAMA_URL = "http://localhost:11434/api/generate"
MODEL = "qwen2.5:1.5b"
PROMPT = "Explain what a load balancer does in two sentences."


def naive_benchmark():
    start = time.perf_counter()
    response = requests.post(
        OLLAMA_URL,
        json={"model": MODEL, "prompt": PROMPT, "stream": False},
    )
    elapsed = time.perf_counter() - start
    data = response.json()
    print(f"Latency: {elapsed:.3f}s")
    print(f"Response: {data['response'][:80]}...")
    return elapsed


if __name__ == "__main__":
    naive_benchmark()

Running this once, as the very first request after starting the benchmark session, produced:

Latency: 6.015s
Response: A load balancer distributes network traffic evenly among multiple servers to imp...

Six seconds. Is that the model’s real latency? Running the exact same script five more times immediately afterward tells a different story:

Latency: 3.000s
Latency: 2.739s
Latency: 2.662s
Latency: 2.644s
Latency: 2.685s

The first call was more than twice as slow as every call after it. If you had run this benchmark exactly once, as most naive benchmarks do, you would have reported a number that is wrong by roughly 2.2x, and you would have had no way to know it. This is the warm-up effect, and it is the first thing any real benchmark harness has to account for.

Step 3: Why the First Request Is Always Slow: Understanding Warm-Up

Before an inference server can generate a single token, it has to get the model’s weights into memory (RAM or VRAM) and initialize the runtime around them. Ollama calls this loading the model, and it happens automatically on the first request to a model, then the model stays resident in memory for a configurable idle period afterward (controlled by the keep_alive parameter) so subsequent requests skip straight to generation.

You can see this directly, because Ollama’s own API response includes a timing breakdown. Sending "keep_alive": 0 forces the model to unload immediately after a request, which lets you deliberately reproduce a cold start on demand instead of waiting for one to happen naturally:

import time
import requests

OLLAMA_URL = "http://localhost:11434/api/generate"
MODEL = "qwen2.5:1.5b"
PROMPT = "Explain what a load balancer does in two sentences."
NS_PER_S = 1_000_000_000


def unload_model():
    requests.post(OLLAMA_URL, json={"model": MODEL, "prompt": "", "keep_alive": 0})
    time.sleep(1)


def timed_request(label):
    start = time.perf_counter()
    response = requests.post(
        OLLAMA_URL,
        json={"model": MODEL, "prompt": PROMPT, "stream": False},
    )
    wall_clock = time.perf_counter() - start
    data = response.json()

    total_duration = data["total_duration"] / NS_PER_S
    load_duration = data["load_duration"] / NS_PER_S
    prompt_eval_duration = data["prompt_eval_duration"] / NS_PER_S
    eval_duration = data["eval_duration"] / NS_PER_S
    eval_count = data["eval_count"]

    print(f"--- {label} ---")
    print(f"wall-clock latency:    {wall_clock:.3f}s")
    print(f"total_duration:        {total_duration:.3f}s")
    print(f"  load_duration:       {load_duration:.3f}s  (model into memory)")
    print(f"  prompt_eval_duration:{prompt_eval_duration:.3f}s  (processing the prompt)")
    print(f"  eval_duration:       {eval_duration:.3f}s  (generating {eval_count} tokens)")
    print(f"  tokens/sec (decode): {eval_count / eval_duration:.1f}")


if __name__ == "__main__":
    unload_model()
    timed_request("COLD START")
    timed_request("WARM (immediately after)")

Real captured output:

--- COLD START ---
wall-clock latency:    5.060s
total_duration:        3.021s
  load_duration:       2.568s  (model into memory)
  prompt_eval_duration:0.123s  (processing the prompt)
  eval_duration:       0.320s  (generating 18 tokens)
  tokens/sec (decode): 56.2

--- WARM (immediately after) ---
wall-clock latency:    2.537s
total_duration:        0.491s
  load_duration:       0.186s  (model into memory)
  prompt_eval_duration:0.018s  (processing the prompt)
  eval_duration:       0.282s  (generating 17 tokens)
  tokens/sec (decode): 60.3

load_duration alone accounts for 2.568 seconds of the cold request, on a small 1.5 billion parameter model. On a larger model, this cost is even higher. The generation speed itself (56.2 versus 60.3 tokens per second) is basically identical between cold and warm; the entire difference is the one-time cost of getting the model into memory. The lesson: always run at least a few warm-up requests and throw away their results before you start measuring. Every benchmark harness in the rest of this tutorial does exactly that.

The Hidden Trap: “localhost” vs “127.0.0.1”

While building the cold-start test above, the numbers did not add up. Look again: the WARM request’s total_duration, Ollama’s own self-reported server-side timing, was only 0.491 seconds. But the wall-clock time measured on the client side was 2.537 seconds. Something was eating roughly two seconds that Ollama itself was not accounting for, and it was happening on every request, warm or cold.

The culprit turned out to be one word: localhost. Every script up to this point used http://localhost:11434. Switching to the literal loopback address, http://127.0.0.1:11434, and running three requests each way, produced this:

localhost : 2.834s
localhost : 2.752s
localhost : 2.723s
127.0.0.1 : 0.450s
127.0.0.1 : 0.447s
127.0.0.1 : 0.460s

Using the hostname localhost instead of the IP address added a consistent, repeatable 2.3 seconds to every single request, regardless of whether the model was warm. That is not model latency at all; it is pure connection overhead, and it would silently corrupt every number in a benchmark run without localhost ever looking suspicious, since the requests still succeed and still return the correct answer.

Diagnosing exactly where the delay came from is worth walking through, because “trust but verify your own tooling” is itself the real lesson here. Testing a raw socket connection directly, bypassing any HTTP library, showed both DNS resolution and the TCP connect itself completing in under a millisecond for localhost:

getaddrinfo('localhost'): 7.0ms -> families returned: ['AF_INET6', 'AF_INET']
  raw AF_INET connect() using 'localhost' as-is: 0.4ms

getaddrinfo('127.0.0.1'): 0.0ms -> families returned: ['AF_INET']
  raw AF_INET connect() using '127.0.0.1' as-is: 0.1ms

So the delay was not DNS resolution and not the raw TCP handshake. The next suspect was proxy auto-detection: on Windows, HTTP client libraries can check system proxy settings before connecting, and that check can be slow. Explicitly disabling proxy handling on the requests session ruled that out too; the delay persisted:

System proxy settings urllib sees: {}

localhost, default requests behavior     : 4.549s
localhost, proxies explicitly disabled    : 2.561s
localhost, session.trust_env = False       : 2.767s

The delay also reproduced identically using Python’s own standard-library urllib.request, with no third-party HTTP library involved at all, which confirmed it was not a quirk of the requests package specifically. The honest conclusion: on this machine, something in the connection setup path adds roughly 2.3 seconds specifically when the hostname string localhost is used, and that overhead is not visible in DNS timing, raw socket timing, or proxy configuration. The exact culprit (likely local network stack or security software behavior particular to this environment) matters less than the actionable fix: use the explicit loopback IP address for local benchmarking, and always sanity-check your benchmark tool’s own overhead against a trivial request before you trust any number it reports. Your environment may not reproduce this exact 2.3 second figure, but you should still run this check, since a client-side artifact like this one will not show up in a server’s self-reported timing fields at all.

Step 4: Measuring Time to First Token the Right Way

Everything so far has measured only end-to-end latency: the time until the entire response is done. TTFT requires something different: streaming the response and timestamping the moment the first chunk arrives, not the moment the last one does. Set "stream": True and read the response line by line, since Ollama streams one JSON object per line:

import time
import json
import requests

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "qwen2.5:1.5b"
PROMPT = "Explain what a load balancer does in two sentences."


def streaming_request():
    t_start = time.perf_counter()
    first_token_time = None
    token_count = 0
    final_stats = None

    with requests.post(
        OLLAMA_URL,
        json={"model": MODEL, "prompt": PROMPT, "stream": True},
        stream=True,
    ) as response:
        for line in response.iter_lines():
            if not line:
                continue
            chunk = json.loads(line)
            if first_token_time is None and chunk.get("response"):
                first_token_time = time.perf_counter()
            if chunk.get("response"):
                token_count += 1
            if chunk.get("done"):
                final_stats = chunk

    t_end = time.perf_counter()
    ttft = first_token_time - t_start
    e2e_latency = t_end - t_start
    eval_count = final_stats["eval_count"]
    eval_duration_s = final_stats["eval_duration"] / 1_000_000_000
    tpot = (e2e_latency - ttft) / (eval_count - 1) if eval_count > 1 else None

    return {
        "ttft": ttft,
        "e2e_latency": e2e_latency,
        "eval_count": eval_count,
        "tpot": tpot,
        "decode_tps": eval_count / eval_duration_s,
    }


if __name__ == "__main__":
    result = streaming_request()
    print(f"TTFT:              {result['ttft']:.3f}s")
    print(f"End-to-end latency:{result['e2e_latency']:.3f}s")
    print(f"Output tokens:     {result['eval_count']}")
    print(f"TPOT:              {result['tpot']*1000:.1f}ms/token")
    print(f"Decode throughput: {result['decode_tps']:.1f} tok/s")

Output:

TTFT:              0.429s
End-to-end latency:1.033s
Output tokens:     32
TPOT:              19.5ms/token
Decode throughput: 53.1 tok/s

The tpot calculation follows vLLM’s own formula exactly: end-to-end latency minus TTFT, divided by output tokens minus one. Note it is computed slightly differently from decode_tps, which comes straight from Ollama’s own eval_count and eval_duration fields; TPOT includes a sliver of client-side and post-generation overhead that Ollama’s internal decode timer does not, which is exactly the kind of small discrepancy worth noticing rather than ignoring.

Step 5: Building a Real Benchmark Harness: Warm-Up, N Samples, and Percentiles

One streaming request is better than one non-streaming request, but it is still one data point. A real harness needs three things: warm-up requests that get discarded, enough measured requests to compute a meaningful distribution, and percentile math instead of just an average.

Python’s standard library statistics module can compute percentiles directly with statistics.quantiles(). Calling it with n=100 divides the data into 100 groups and returns 99 cut points; index 49 of that list approximates the 50th percentile (the median), index 89 approximates the 90th, and so on:

import time
import json
import statistics
import requests

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "qwen2.5:1.5b"
PROMPT = "Explain what a load balancer does in two sentences."


def one_request():
    t_start = time.perf_counter()
    first_token_time = None
    final_stats = None

    with requests.post(
        OLLAMA_URL,
        json={"model": MODEL, "prompt": PROMPT, "stream": True},
        stream=True,
    ) as response:
        for line in response.iter_lines():
            if not line:
                continue
            chunk = json.loads(line)
            if first_token_time is None and chunk.get("response"):
                first_token_time = time.perf_counter()
            if chunk.get("done"):
                final_stats = chunk

    t_end = time.perf_counter()
    return {
        "ttft": first_token_time - t_start,
        "e2e_latency": t_end - t_start,
        "output_tokens": final_stats["eval_count"],
    }


def percentile(data, pct):
    cuts = statistics.quantiles(data, n=100, method="inclusive")
    return cuts[pct - 1]


def run_benchmark(warmup=3, measured=30):
    print(f"Running {warmup} warm-up requests (discarded)...")
    for _ in range(warmup):
        one_request()

    print(f"Running {measured} measured requests...")
    results = [one_request() for _ in range(measured)]

    ttfts = [r["ttft"] for r in results]
    latencies = [r["e2e_latency"] for r in results]

    print(f"n = {measured}")
    print(f"{'metric':<12}{'mean':>8}{'p50':>8}{'p90':>8}{'p95':>8}{'max':>8}")
    for name, series in [("TTFT (s)", ttfts), ("Latency (s)", latencies)]:
        print(
            f"{name:<12}"
            f"{statistics.mean(series):>8.3f}"
            f"{percentile(series, 50):>8.3f}"
            f"{percentile(series, 90):>8.3f}"
            f"{percentile(series, 95):>8.3f}"
            f"{max(series):>8.3f}"
        )


if __name__ == "__main__":
    run_benchmark()

Real output from 30 measured requests, after 3 discarded warm-up requests:

n = 30
metric          mean     p50     p90     p95     max
TTFT (s)       0.189   0.187   0.207   0.214   0.222
Latency (s)    0.496   0.480   0.516   0.620   0.791

Look at the latency row. The mean is 0.496 seconds. But the 95th percentile is 0.620 seconds, and the slowest request took 0.791 seconds, nearly 65 percent longer than the median. If you only reported the mean, you would be hiding the fact that a meaningful share of requests took noticeably longer, and in a real product, those are the requests users complain about. A mean is a single number that describes the middle of a distribution; it cannot describe the tail, and the tail is usually what matters.

One honest caveat: percentiles need enough samples to mean much. A p99 computed from 30 requests is really just an estimate of “the slowest request or two,” not a statistically reliable tail estimate; you generally want a hundred or more samples before you trust a p99 number, and correspondingly more for p999. With 30 samples, p50 and p90 are reasonably trustworthy; treat p95 here as a rough guide and nothing tighter.

Step 6: The Throughput vs Latency Tradeoff Under Concurrency

Every measurement so far has been sequential: one request finishes before the next one starts. Real traffic is not like that; multiple users hit the same server at the same time. Testing that changes the picture entirely. This harness reuses one_request() and percentile() unchanged from Step 5, along with the same mean/p50/p90/p95/max table format, but runs 20 requests sequentially, then runs 20 more using a thread pool with 5 requests in flight at once:

from concurrent.futures import ThreadPoolExecutor
# ... one_request() and percentile() as defined above ...

if __name__ == "__main__":
    N = 20
    for _ in range(3):
        one_request()  # warm-up

    t0 = time.perf_counter()
    seq_results = [one_request() for _ in range(N)]
    seq_wall = time.perf_counter() - t0

    CONCURRENCY = 5
    t0 = time.perf_counter()
    with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
        conc_results = list(pool.map(one_request, range(N)))
    conc_wall = time.perf_counter() - t0

Real results, same model, same prompt, same machine:

--- SEQUENTIAL (concurrency=1, n=20) ---
wall-clock for batch: 9.882s
aggregate throughput: 38.5 tok/s (all requests combined)
metric          mean     p50     p90     p95     max
TTFT (s)       0.183   0.184   0.197   0.197   0.200
Latency (s)    0.494   0.477   0.540   0.562   0.687

--- CONCURRENT (concurrency=5, n=20) ---
wall-clock for batch: 7.344s
aggregate throughput: 56.0 tok/s (all requests combined)
metric          mean     p50     p90     p95     max
TTFT (s)       1.330   1.302   1.903   1.920   1.953
Latency (s)    1.668   1.666   2.215   2.236   2.380

Two things happened, in opposite directions, at the same time. The whole batch of 20 requests finished faster under concurrency (7.344s versus 9.882s), and aggregate throughput went up (56.0 versus 38.5 tokens per second). By a throughput-only measurement, concurrency looks like a clear win. But look at individual TTFT: the median went from 0.184 seconds to 1.302 seconds, a 7x increase. Anyone whose request landed behind four others in the queue waited over a second just to see the first word appear.

This is not a bug; it is a direct, documented consequence of how Ollama handles concurrency. Ollama’s own FAQ states that OLLAMA_NUM_PARALLEL, “the maximum number of parallel requests each model will process at the same time,” defaults to 1. You can read this in Ollama’s official FAQ. With the default configuration, five concurrent requests do not run in parallel at all; they queue and get processed one at a time, and each request’s measured TTFT includes however long it waited in that queue before its own generation even began. This is exactly why “aggregate throughput” and “the latency any individual user experiences” are different metrics that can move in opposite directions, and why a benchmark that only reports one of them is only telling you half the story. For a deeper look at this exact tension in production serving systems, see this site’s earlier piece on why LLM serving benchmarks need goodput, not just throughput.

Step 7: The Reasoning-Model Trap: When “First Token” Is Not the Answer

The final trap only shows up with reasoning models, and it is the most dramatic one in this entire tutorial. Models like qwen3.5 can emit internal “thinking” tokens before producing their visible answer; Ollama streams these separately, in a thinking field distinct from the response field. A benchmark that naively times “the first streamed chunk of any kind” will measure the thinking phase, not the moment a user actually sees an answer.

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "qwen3.5:4b"
PROMPT = "Explain what a load balancer does in two sentences."


def timed_request(think):
    t_start = time.perf_counter()
    first_thinking_time = None
    first_response_time = None
    final_stats = None

    with requests.post(
        OLLAMA_URL,
        json={"model": MODEL, "prompt": PROMPT, "stream": True, "think": think},
        stream=True,
    ) as response:
        for line in response.iter_lines():
            if not line:
                continue
            chunk = json.loads(line)
            if chunk.get("thinking") and first_thinking_time is None:
                first_thinking_time = time.perf_counter()
            if chunk.get("response") and first_response_time is None:
                first_response_time = time.perf_counter()
            if chunk.get("done"):
                final_stats = chunk

    t_end = time.perf_counter()
    return {
        "ttft_thinking": (first_thinking_time - t_start) if first_thinking_time else None,
        "ttft_visible_answer": (first_response_time - t_start) if first_response_time else None,
        "e2e_latency": t_end - t_start,
        "eval_count": final_stats["eval_count"],
    }


if __name__ == "__main__":
    for _ in range(2):
        timed_request(True)  # warm up

    print("--- think=True (qwen3.5 default) ---")
    r = timed_request(True)
    print(f"Time to FIRST THINKING token:  {r['ttft_thinking']:.3f}s")
    print(f"Time to first VISIBLE answer:  {r['ttft_visible_answer']:.3f}s")
    print(f"Total tokens (thinking+answer):{r['eval_count']}")
    print(f"E2E latency:                   {r['e2e_latency']:.3f}s")

    print("--- think=False ---")
    r2 = timed_request(False)
    print(f"Time to first VISIBLE answer:  {r2['ttft_visible_answer']:.3f}s")
    print(f"Total tokens:                  {r2['eval_count']}")
    print(f"E2E latency:                   {r2['e2e_latency']:.3f}s")

Real, unedited output, on the same machine, same prompt:

--- think=True (qwen3.5 default) ---
Time to FIRST THINKING token:  0.556s
Time to first VISIBLE answer:  43.952s
Total tokens (thinking+answer):787
E2E latency:                   46.817s

--- think=False ---
Time to first VISIBLE answer:  0.617s
Total tokens:                  44
E2E latency:                   3.151s

With thinking enabled, the model started producing tokens (its internal reasoning) after just 0.556 seconds, a number that looks perfectly healthy. But the visible answer, the text an actual user would read, did not begin until 43.952 seconds had passed: a 79x difference between “time to first token” and “time to first token a human being can actually see.” In this run, the model generated 787 tokens total to answer a two-sentence question, and the overwhelming majority of them were invisible reasoning the user never sees streamed. Disabling thinking ("think": false) drops the visible-answer latency back down to 0.617 seconds, a number consistent with the non-reasoning model tested throughout this tutorial. Re-running this same test produces different exact numbers each time (LLM output length is not deterministic), but the shape of the result, a first visible token dozens of seconds later than the first thinking token, is consistently reproducible; an earlier run of this same code measured a 94x gap (0.585s versus 55.105s over 960 tokens), just as dramatic.

Sanity-checking the per-token generation speed confirms the story: 787 tokens over 46.817 seconds is about 16.8 tokens per second, and 44 tokens over 3.151 seconds is about 14.0 tokens per second, close enough to the same underlying decode speed. The 79x latency difference is not the model getting slower; it is 743 extra invisible tokens getting generated before the reader sees a single word. If you are benchmarking or deploying a reasoning model, always distinguish “first token of any kind” from “first token a user actually sees,” and be deliberate about whether hidden reasoning tokens should count in a latency budget at all.

Common Mistakes and Gotchas

Everything above was a real, reproduced trap. Collected in one place, as a checklist:

• Trusting a single measurement. One request tells you about one request; it cannot tell you what typical or worst-case behavior looks like.
• Not discarding warm-up requests. The first request after a model loads pays a one-time cost (2.568 seconds of pure model-loading time in this tutorial’s own measurements) that has nothing to do with steady-state performance.
• Using a hostname instead of an explicit loopback IP for local benchmarks. On this machine, localhost added a flat 2.3 seconds of connection overhead to every request compared to 127.0.0.1, silently, with no error and no visible symptom other than the numbers being wrong.
• Reporting only the mean. A mean can look perfectly healthy while a meaningful fraction of requests are noticeably slower; percentiles, especially p95 and above, are what expose that.
• Claiming a percentile your sample size cannot support. A p99 from 30 samples is not a real p99; it takes on the order of a hundred or more samples before tail percentiles stabilize into something you can trust.
• Benchmarking only at concurrency=1. A benchmark run with no concurrent load will not show you the queueing effects real traffic produces; this tutorial’s own numbers show median TTFT jumping 7x under five-way concurrency with Ollama’s default OLLAMA_NUM_PARALLEL=1 setting.
• Treating “first streamed token” as “first visible token” for reasoning models. With thinking enabled, this tutorial measured a gap of 79x to 94x between the two across repeated runs. If you don’t separate them, your TTFT number will describe how fast the model started thinking, not how fast a user got an answer.

How to Verify Your Benchmark Is Actually Working

Before trusting any benchmark harness, including the one built in this tutorial, run through this checklist:

1. Run it twice and compare. The two runs should produce broadly similar percentiles (within normal noise). Wildly different results between runs usually means something external is interfering (background processes, thermal throttling, another program using the same server).
2. Confirm the warm-up actually ran first. Print or log the warm-up requests separately so you can see they happened and were excluded, not accidentally included in your measured set.
3. Sanity-check your percentiles are internally consistent. By definition, p50 should be less than or equal to p90, which should be less than or equal to p95, which should be less than or equal to the maximum. If they are not in that order, there is a bug in your percentile code, not in the model.
4. Cross-check your computed tokens-per-second against the server’s own numbers. Ollama’s response includes eval_count and eval_duration directly; your own throughput calculation should land close to eval_count / (eval_duration / 1e9). A large mismatch means your timing code, not the model, is the thing to debug.
5. Test your connection baseline before you test the model. Time a trivial request (or even just a TCP connect) against your target before running the real benchmark, exactly like the localhost versus 127.0.0.1 test in Step 3. If that baseline number looks high, fix your connection method first; otherwise every subsequent number is contaminated.

Next Steps

The harness built here generalizes well beyond a single local model:

• Point the same scripts at two different quantization levels of the same model (or two different model sizes) and compare their p50 and p95 TTFT and throughput side by side.
• Compare a local Ollama deployment against a hosted API using the same percentile methodology, to make an honest cost-versus-latency decision instead of an anecdotal one.
• Wire a simplified version of this harness into a CI pipeline as a performance regression gate: fail the build if p95 latency regresses beyond a threshold after a model or configuration change.
• If you are building an agent or chat application on top of inference, see this site’s tutorials on cutting AI agent latency with streaming and parallel tool calls, diagnosing a low prompt cache hit rate, and building a multi-tier LLM router with automatic fallback, all of which build on the same TTFT and percentile fundamentals covered here.