Prefill/Decode Disaggregation Shows Why LLM Inference Needs Separate GPU Pools

Every request to a large language model runs through two phases that behave nothing alike. The model first reads the prompt in one dense burst of matrix computation (prefill), then writes the answer one token at a time (decode). Most production deployments still run both phases on the same GPU, and a growing number of the largest LLM operators have concluded that is the wrong default at scale.

Moonshot AI’s Mooncake, DeepSeek’s inference stack, and NVIDIA’s own Dynamo framework all now run prefill and decode on physically separate GPU pools, linking them by shipping the model’s key-value cache over the network between phases. A July 15 DigitalOcean engineering write-up lays out why the split is becoming standard practice at scale, and when it is not worth the added hardware and complexity.

The Problem: One GPU, Two Incompatible Jobs

Prefill and decode stress opposite parts of a GPU. Prefill processes every token of the input prompt at once, in parallel, which is the kind of dense matrix multiplication GPUs are built for: compute cores run near saturation while memory bandwidth sits mostly unused. Decode does the opposite. Generating each output token requires rereading the model’s full set of weights plus the growing key-value (KV) cache, the accumulated memory of the conversation so far, from GPU memory. That per-token computation is small, so memory bandwidth becomes the bottleneck while compute cores sit mostly idle.

Run both phases on one shared GPU, and they compete. A single long prompt occupying the compute cores for prefill delays every other user’s in-flight decode step: the mechanism DigitalOcean’s write-up calls the “prefill tax,” latency that grows with how many tokens the model has to read before it starts responding.

What Disaggregation Actually Changes

Prefill/decode disaggregation is a serving architecture that assigns each phase to its own dedicated pool of GPUs. Prefill runs on one set of GPUs; once it finishes processing a prompt, the model’s KV cache for that request is transferred over the network to a GPU in the decode pool, which then generates the response token by token.

The tradeoff is explicit: extra hardware, a network transfer for every request, and real operational complexity, in exchange for eliminating the interference between the two phases. That is a good trade only once a deployment is big enough, and busy enough, for the interference to actually cost something.

The KV Cache Handoff

The KV cache is the part of a transformer model’s internal state that lets it avoid recomputing attention over the entire prompt for every new token; it is why decode can generate one token at a time instead of re-reading the whole conversation from scratch each time. In a disaggregated setup, that cache has to leave the GPU where it was built and arrive, intact, on a different machine, fast enough that the handoff does not itself become the bottleneck. Production systems solve this with dedicated high-throughput transfer engines rather than routing the cache through general-purpose networking, which is most of what Mooncake, described below, actually is.

Three Production Systems Already Split Prefill From Decode

Moonshot AI’s Mooncake Serves Kimi

Mooncake is the serving platform behind Kimi, the LLM product from Chinese AI lab Moonshot AI. Moonshot describes it in the project’s GitHub repository as a “KVCache-centric disaggregated architecture for LLM serving,” built around a dedicated transfer engine for moving KV cache between prefill and decode nodes rather than treating that transfer as an afterthought. The design won the Best Paper award at FAST 2025, the 23rd USENIX Conference on File and Storage Technologies. According to that paper, Mooncake increases effective request capacity by 59 to 498 percent over baseline methods in long-context scenarios, and in production it lets Kimi handle 115 percent more requests on Nvidia A800 clusters and 107 percent more on H800 clusters compared to Moonshot’s previous serving system. Mooncake now runs across thousands of nodes processing more than 100 billion tokens a day, and it has since been adopted as a KV-cache transfer backend by other serving engines, including vLLM and SGLang.

DeepSeek Runs EP32 Prefill Against EP144 Decode

DeepSeek published some of the clearest public numbers on what disaggregation looks like at full scale. In a February 2025 engineering write-up covering its V3 and R1 models, part of what the company called Open Source Week, DeepSeek confirmed it has “adopted a prefill-decode disaggregation architecture” and runs a different degree of parallelism in each phase: prefill uses expert parallelism across 32 GPUs (EP32) with each deployment unit spanning 4 nodes, while decode scales that out to EP144 across 18 nodes. All of it ran on Nvidia H800 GPUs, the reduced-interconnect variant Nvidia sells into the China market.

The same write-up disclosed real operating figures: over a 24-hour window (February 27 to 28, 2025), V3 and R1 inference occupied an average of 226.75 nodes at 8 H800 GPUs each, for a total infrastructure cost of $87,072 at an assumed $2 per GPU-hour. Had every token been billed at DeepSeek-R1’s public API pricing, DeepSeek calculated the same infrastructure would have generated $562,027 in theoretical daily revenue, a 545 percent cost-to-revenue margin. The company was explicit that its real revenue runs far lower, since R1 pricing does not apply to V3 traffic, free web and app usage is not monetized, and nighttime discounts apply automatically.

NVIDIA Dynamo Makes the Split a Standard Feature

NVIDIA’s answer is Dynamo, which its project repository describes as an open source, datacenter-scale distributed inference serving framework. Dynamo does not replace inference engines like vLLM, SGLang, or TensorRT-LLM; it sits above them as an orchestration layer, coordinating disaggregated prefill/decode serving, KV-aware request routing, multi-tier KV caching, and automatic scaling across a multi-node deployment. Because it ships as open source infrastructure rather than a single vendor’s hosted product, Dynamo is one of the reasons disaggregation has moved from research technique to something teams can adopt without building the orchestration layer themselves.

Disaggregation Is Not a Default, It Is a Threshold

None of the three systems above treat disaggregation as something every deployment needs. DigitalOcean’s write-up frames the decision around three workload signals, and all three point the same direction: disaggregation earns its complexity at scale, not by default.

Traffic Volume: The Win Shows Up at Saturation

A small deployment with a handful of GPUs and sporadic requests does not benefit much; running prefill and decode on the same GPUs is simpler, cheaper, and easier to operate. The benefit appears once a fleet is busy enough that prefill and decode are constantly competing for the same compute and memory bandwidth. Separating them into dedicated pools is what recovers the throughput that competition was costing.

Latency Consistency: Interactive Workloads Feel It Most

Batch workloads, like document processing or offline analysis, tolerate an occasional slow request without anyone noticing. Interactive workloads such as chatbots and coding assistants do not: a single long prompt occupying a shared GPU can visibly delay every other user’s streaming response. The more a product depends on predictable, fast time-to-first-token, the more disaggregation’s isolation is worth its added complexity.

Prompt Length: Long-Context Workloads Gain the Most

Short prompts finish prefill quickly no matter the architecture, so separating the phases buys little. Workloads dominated by long documents, retrieval-augmented generation, or long conversation histories spend far more wall-clock time in prefill, which is exactly the phase disaggregation insulates from decode’s interference. The longer and more variable the prompts, the larger the gain.

Matching Hardware to the Phase

DigitalOcean’s own Gradient AI Inference Cloud shows what the split looks like in hardware terms: prefill runs on pools of Nvidia H100 GPUs, and decode runs on pools of Nvidia H200 GPUs. That assignment follows directly from what each phase needs. Nvidia’s own H200 product page describes the H200 as offering “nearly double the capacity” of the H100 (141GB of HBM3e memory versus the H100’s HBM3) and about 1.4 times the memory bandwidth. Decode is the phase that repeatedly rereads model weights and an ever-growing KV cache, so it benefits directly from the H200’s larger, faster memory. Prefill’s dense, parallel matrix computation is compute-bound rather than memory-bound, so it runs efficiently on H100 capacity without paying for memory it will not use.

Buying and provisioning to that split, rather than sizing every GPU in the fleet for whichever phase is harder, is the practical payoff disaggregation offers once a deployment has grown large enough to need it.

The Takeaway

Prefill/decode disaggregation is not a new algorithm or a smarter model; it is an operational decision about where to spend hardware and network complexity in exchange for throughput and latency at scale. Mooncake, DeepSeek’s inference stack, and NVIDIA Dynamo all reach the same architecture from different directions, which is a reasonably strong signal that the pattern is becoming a standard part of the production LLM serving stack rather than a one-off optimization. Teams running small or bursty deployments still have the simpler option available; the three signals above (traffic volume, latency sensitivity, and prompt length) are what determine which side of that line an actual deployment falls on.