Why LLM Serving Benchmarks Need Goodput, Not Just Throughput

The default way to benchmark an LLM serving setup is a single number: throughput, or how many requests per second the system can push through. It is easy to measure, easy to compare, and it lines up with the number finance cares about most, dollars per request. So teams tune batching, raise concurrency, and squeeze the GPU until that number climbs. Graziano Casto, an engineer at the infrastructure optimization company Akamas, ran into what happens next while tuning vLLM on a single GPU: throughput kept climbing while the service itself quietly got worse. He wrote up the gap in a CNCF Blog member post published July 20, and the pattern he found applies well beyond his own setup.

A Kitchen Analogy for a Real Problem

Casto frames the gap with a restaurant analogy. Throughput is the number of plates that leave the kitchen every hour: it says the kitchen is busy, but nothing about whether the food arrived hot, on time, or at the right table. A kitchen can double its plates per hour and still send out a room full of cold, late dinners. The metric that actually captures what a user experiences is goodput, the number of completed requests per second that also meet defined latency targets.

For LLM serving, those targets are usually two numbers. Time to first token (TTFT) is the wait before anything appears on screen. Time per output token (TPOT) is the pace at which the rest of the answer streams out after that. A request that finishes but blows past either target still counts toward throughput; it does not count toward goodput.

The Test Setup: A Deliberately Modest GPU

Every part of Casto’s benchmark runs on infrastructure most teams already have access to. He served Qwen2.5-7B with vLLM on a single NVIDIA A10G GPU with 24 GB of memory inside an Amazon EKS cluster, generated load with GuideLLM, and watched the system through Prometheus and Grafana, with GPU-level metrics coming from NVIDIA’s DCGM exporter. GuideLLM is worth a second look here: it describes itself as an “SLO-aware Benchmarking and Evaluation Platform for Optimizing Real-World LLM Inference,” originally released by Neural Magic and now maintained under the vLLM project on GitHub. A benchmarking tool built around service-level objectives is not designed to hand back a bare throughput number in the first place, which fits Casto’s argument closely.

Casto picked the A10G on purpose. It is not the GPU a team would choose for a production-grade LLM service, so the absolute numbers in his tests look modest next to what an H100 cluster could do. But the point of the exercise was never the ceiling; it was the shape of the trade-off between throughput and latency, and Casto argues that shape holds regardless of how large the GPU gets. The full setup, Kubernetes manifests included, is published on GitHub so other teams can reproduce the runs or point the same benchmark at their own models and real traffic instead of his synthetic load.

Three Workloads, Three Different Breaking Points

Casto ran the same infrastructure against three kinds of traffic, because the shape of a workload decides which latency target actually bites first: a chatbot workload (about 485 prompt tokens and 121 output tokens, where a tight TTFT and a medium TPOT matter most), a reasoning workload (about 400 prompt tokens and 4,000 output tokens, where TPOT matters most and TTFT can be loose), and an agentic workload that chains many short calls together (about 512 prompt tokens and 128 output tokens per call, where a slow per-call TTFT adds up across the chain).

Chatbot: A Faster Number That Hid a Slower Stream

For the chatbot workload, Casto’s search had one job: maximize combined token throughput (prefill plus decode) under a hard constraint that average TTFT stay at or below 1.5 seconds. To avoid scoring a lucky spike or an unlucky dip, he added a windowing rule that only scored a configuration after eight consecutive steady data points, since a live throughput metric jitters as individual requests start and finish.

The winning configuration, which Casto labeled Exp 20, beat an earlier configuration, Exp 1, by a wide margin: about 50 percent more combined throughput, with both landing at nearly the same TTFT, right at the edge of the 1.5 second ceiling. On a throughput chart alone, Exp 20 looks like the easy winner. The bill showed up somewhere the search was never told to protect: Exp 1’s p95 TPOT sat at about 50 milliseconds, while Exp 20’s sat at about 494 milliseconds, nearly ten times higher. The search had been told to protect the first token; it had not been told to protect the pace of the stream that followed, so it spent that budget without being asked to.

Agentic: Tied on Throughput, Not on Experience

In the agentic study, the two best configurations Casto found were close to tied on raw throughput: 6.20 requests per second against 6.11. Their p95 TPOT told a different story: 682 milliseconds against 902 milliseconds, a gap of about 220 milliseconds. Two configurations that look nearly identical on a throughput dashboard produce a noticeably different stream once a user is inside one of those many chained calls.

Reasoning: Long Answers Change the Winning Config

In the reasoning study, where each answer ran to about 4,000 tokens, the configuration with the best raw throughput was not the one Casto would have picked once latency mattered. Asking the same automated search to respect a tighter inter-token target pushed the winning configuration to something entirely different.

Three studies, three different ways of asking the same underlying question, and Casto found the same answer each time: “a throughput number with nothing else attached does not tell you which configuration to run.”

Three Lessons for Anyone Running Inference in Production

Casto draws three conclusions from the pattern, plus a fourth he flags as a hypothesis rather than something his own tests confirmed.

First, a throughput number with no SLO attached is, in his words, “marketing, not engineering.” The honest target is goodput: requests per second that actually met the latency targets that matter for that workload. Anything else is counting cold plates.

Second, the configuration with the best goodput is rarely the one an engineer would guess. In Casto’s reasoning study, two settings that looked almost identical ended up far apart in practice, and a nearby setting failed outright. He describes the search space as “full of small cliffs,” which is exactly why hand tuning tends to stop at the first number that looks good rather than the best one actually available. His answer is to replace hand tuning with an automated, data-driven search that tests many configurations against real targets rather than a handful of guesses.

Third, none of this holds still. Traffic patterns, prompt shapes, and the models themselves change over time, and the best configuration moves with them. That means the search has to run on an ongoing basis, not once at launch.

The fourth point is a caveat Casto raises but did not test directly: he changed only three serving knobs in these studies, and the gap between configurations was already this wide. The same trade-off, he argues, likely lives in the layers he held fixed: GPU settings, the runtime, and scheduling. The largest gains tend to come from tuning the whole stack together rather than one layer at a time, since the layers interact with each other. His practical suggestion is to set TTFT and TPOT targets first, then search for the configuration that delivers the most throughput inside those targets, rather than searching for throughput first and checking latency afterward.

Worth Noting: The Fix Casto Recommends Is Also What Akamas Sells

It is worth reading that fourth lesson alongside who wrote it. Akamas describes itself on its own site as an “Autonomous Kubernetes Optimization Platform” that “tunes the serving engine, runtime, GPU, and infrastructure together” for AI and GPU workloads, using what it calls “patented reinforcement learning” to optimize an entire stack instead of one layer at a time. That is close to word for word what Casto’s fourth lesson recommends doing. None of that makes the underlying data wrong: the gap between Exp 1 and Exp 20 is a measured result, not a marketing claim, and the case for goodput over bare throughput does not depend on whose product benefits from it. But a benchmark whose own lessons conclude that engineers should stop hand tuning and automate the whole stack together is also, unavoidably, a pitch for the exact category of product its author’s company sells.

The Takeaway

Whatever tool a team uses to act on it, Casto’s core finding does not depend on his employer’s product. A throughput number by itself cannot tell an engineering team which configuration to run in production, only whether the GPU is busy. The full benchmark setup is public on GitHub, so any team running vLLM, or any other serving engine, can point the same method at its own models and its own traffic and answer the question that actually matters: is this configuration serving users well, or is it just staying busy?