Prefill-Decode Disaggregation: The Splitwise AI Inference Architecture

Artificial intelligence is currently trapped in a hardware paradox. When you send a 5,000-word prompt to a Large Language Model (LLM), the GPU executes two completely different tasks. First, it “reads” your entire prompt simultaneously in a massive, mathematically explosive burst of energy. Second, it “writes” the response out, painfully generating exactly one word at a time, spending most of its time sitting idle waiting for its memory chips to catch up. By forcing the same $30,000 NVIDIA GPU to perform both tasks, cloud providers are essentially using a Ferrari to drag a plow; the hardware is never operating in its optimal regime.

Why should you care right now? Because inference engineering has finally solved this bottleneck by severing the process in half. Driven by seminal 2024 architectures like DistServe and Splitwise, tech giants are executing “prefill-decode disaggregation.” They are physically separating the “reading” brain from the “talking” brain, putting them on entirely different server clusters connected by ultra-high-speed fiber optics. By matching the specific mathematical workload to the exact silicon profile it requires, disaggregation effectively doubles overall token throughput, redefining the unit economics of generative AI deployment for the entire enterprise sector.

What is Prefill-Decode Disaggregation?

Prefill-decode disaggregation is an LLM inference architecture that physically separates the compute-bound prompt processing phase (prefill) from the memory-bound token generation phase (decode). By routing these tasks to distinct, specialized GPU pools connected via high-speed networking, the system eliminates hardware resource contention, drastically improving token throughput and minimizing latency.

At a Glance

• Concept: Splitting a chatbot’s brain in two. One cluster of GPUs exclusively reads incoming questions; another cluster exclusively types out the answers.
• Why it matters: It stops the slow typing process from getting interrupted every time a new, massive question hits the server, increasing the total capacity of an AI data center by over 40%.
• Who uses it: MLOps teams at hyperscalers (AWS, Azure) and frontier AI labs running massive open-weight models on frameworks like vLLM, NVIDIA Dynamo, and Mooncake.
• Biggest takeaway: Moving the data between the two brains requires insane network speeds. The architecture relies entirely on RDMA (Remote Direct Memory Access) to beam gigabytes of memory directly from one GPU to another without touching the main computer processor.

In Simple Words

Imagine a highly popular, fast-food burger kitchen.

In a Colocated (Traditional) System, one chef does everything. He takes the complex custom order from the customer, memorizes it, and then slowly flips the burgers one by one. While he is flipping the burgers, he cannot take a new order. The line out the door gets completely stalled.

In a Disaggregated System, you split the kitchen. You have one hyper-fast “Prefill Chef” at the front whose only job is to take massive, complex orders and instantly summarize them onto a ticket. He then slides that ticket down a wire to the back of the kitchen, where a dedicated “Decode Chef” just stands there flipping burgers non-stop.

By splitting the tasks, the front chef never gets bogged down flipping meat, and the back chef never has to stop cooking to talk to a customer. The restaurant serves twice as many people using the exact same amount of kitchen space.

Why This Matters

For Cloud Architects, MLOps Engineers, and AI Hardware VCs, disaggregation solves the Utilization Asymmetry Problem.

The math of AI inference is shifting. In 2023, users asked short questions (“Write a poem”). In 2026, agentic AI systems are digesting 100,000-token PDFs and entire software codebases. The “prefill” phase (reading the input) is now frequently consuming 50% to 80% of total GPU compute time.

If you run prefill and decode on the same GPU, the massive math required to digest that 100,000-token PDF instantly freezes the GPU. Any other users who are waiting for their next token to be generated (decode) will suddenly experience a massive lag spike. Their chatbot stops typing.

Disaggregation protects the decoders. By isolating the prefill phase on a dedicated “compute-heavy” GPU cluster, the “memory-heavy” decode GPUs can run smoothly and continuously, guaranteeing strict Service Level Agreements (SLAs) for enterprise clients requiring ultra-low-latency real-time voice and text generation.

Heterogeneous Hardware Provisioning in AI Data Centers

Disaggregation allows for Heterogeneous Hardware Provisioning.

Historically, MLOps teams had to buy the most expensive, top-tier GPUs (like the NVIDIA H100) because they needed a chip that was decent at both heavy math and fast memory. Disaggregation breaks this monopoly.

Because the prefill phase only cares about raw math (TFLOPS), you can run your prefill clusters on older, compute-heavy GPUs or cheaper custom ASICs (like Google TPUs or AWS Inferentia). Because the decode phase only cares about memory speed (Bandwidth), you can run your decode clusters on specialized hardware built explicitly with massive High-Bandwidth Memory (HBM3e) stacks. You stop paying for silicon you aren’t using.

How Prefill-Decode Disaggregation Works

Physically splitting a neural network’s thought process across two different machines requires a flawlessly synchronized data handoff. Here is the first-principles breakdown of the architecture.

1. The Fundamental Problem: Contention

During prefill, the GPU processes all prompt tokens in parallel using massive matrix multiplications. During decode, the GPU generates one token, feeds it back into the model, and generates the next. Decode is autoregressive; it relies entirely on a massive memory file called the Key-Value (KV) Cache to remember the context of the conversation. When both phases share a GPU, they fight over the same limited VRAM and scheduling priority.

2. The Core Mechanism: Physical Isolation

The system assigns incoming requests to a “Prefill Worker.” This GPU executes the heavy parallel math on the user’s prompt. As it reads the prompt, it builds the KV Cache (the mathematical summary of the text).

3. Technical Depth: The KV Cache Transfer

Once the prefill is complete, the massive KV Cache must be moved to the “Decode Worker” before the first word can be generated. This is the ultimate bottleneck. If you send a 10-gigabyte KV Cache over a standard TCP/IP network, the delay will be measured in seconds, ruining the user experience.

4. Technical Depth: RDMA and NIXL

To solve this, advanced frameworks (like NVIDIA Dynamo) use Remote Direct Memory Access (RDMA) over InfiniBand or RoCE v2 networks. RDMA completely bypasses the CPU and the operating system of both servers. The networking card on the Prefill server reaches directly into its GPU’s VRAM, grabs the KV Cache, and blasts it over the fiber-optic cable straight into the VRAM of the Decode GPU. This zero-copy transfer happens in single-digit milliseconds.

5. Real-World Consequences: Continuous Batching Optimization

Once the KV cache arrives, the Decode Worker adds it to its continuous batching queue. Because the Decode Worker is no longer being interrupted by new, massive prompts, it can pack its memory entirely full of active decode requests. This pushes the decode GPU’s SM (Streaming Multiprocessor) utilization significantly higher, generating a smooth, relentless stream of output tokens.

LLM Inference Simulator

Commercial Deployments of Disaggregated Inference

The speed of adoption signals a massive paradigm shift. By 2026, disaggregation transitioned from academic papers into the default deployment architecture for frontier models.

High-Throughput API Providers: Companies offering “LLM-as-a-Service” (like Together AI, Fireworks, or Perplexity) operate on brutal razor-thin margins. By deploying disaggregated architectures via frameworks like vLLM and SGLang, they optimize their Time-to-First-Token (TTFT) by up to 2.3x. A fast TTFT gives the illusion of instant intelligence to the end-user, while the protected decode pool maximizes the total tokens generated per dollar of hardware.

Mooncake’s KVCache Centric Architecture: Kimi (Mooncake) pioneered an architecture that treats the KV Cache as the center of gravity, rather than the GPUs. Instead of naively transferring caches after prefill, Mooncake utilizes a global “conductor.” This orchestrator proactively schedules prefill and decode tasks on nodes that already hold relevant chunks of the KV Cache in their local VRAM, reducing the dreaded RDMA network transfer overhead by 35% to 50%.

Dynamic Intra-GPU Disaggregation: For smaller enterprises that cannot afford a massive, multi-rack InfiniBand cluster, platforms are implementing Intra-GPU disaggregation. Frameworks dynamically partition a single massive GPU (like an NVIDIA B200), reserving 30% of the SM cores strictly for prefill and 70% strictly for decode. While it doesn’t offer the scalability of physical separation, it effectively isolates the workloads at the silicon level, preventing prefill math from freezing the decode stream.

Economic & Strategic Impact

The core strategic consequence of disaggregation is the commoditization of routing layers.

Just as Kubernetes abstracted away the physical server for cloud computing, advanced disaggregated routing engines are abstracting away the GPU for AI. The intelligence of the system no longer lies solely in the weights of the neural network; it lies in the load-balancer that decides exactly which GPU gets which piece of the math.

If an enterprise can flawlessly route prefill, execute RDMA transfers, and load-balance decode streams, they can build a virtual supercomputer out of commodity, mismatched GPUs. They do not need a perfectly homogenous cluster of $40,000 chips. This fundamentally threatens the hardware premium commanded by dominant silicon vendors, empowering infrastructure software to bridge the gap in hardware capability.

Advantages

• Zero Interference: Completely prevents massive prompt-processing math from stalling the generation of tokens for other users, guaranteeing smooth, human-like typing speeds.
• Hardware Optimization: Allows architects to buy compute-heavy ASICs for the prefill pool and memory-heavy silicon for the decode pool, drastically lowering Total Cost of Ownership (TCO).
• Independent Scaling: If an application features massive input prompts but short answers (e.g., document summarization), operators can scale up the prefill cluster without wasting money over-provisioning decode nodes.

Limitations

• The RDMA Network Tax: The architecture relies entirely on the network. Moving a 20GB KV Cache between servers takes time. If the InfiniBand network is congested, the Time-to-First-Token (TTFT) actually increases compared to a monolithic setup.
• Multi-Turn Chat Inefficiency: In a long back-and-forth chat session, traditional disaggregation requires repeatedly sending the growing KV Cache back and forth between the prefill and decode nodes.
• Complexity Overkill: For small local models (under 8B parameters) or deployments with low concurrent user traffic, the engineering overhead of managing side-channel communications (NIXL) and dedicated routing queues is entirely unnecessary.

Takeaway: Disaggregation trades a compute bottleneck for a networking bottleneck. You are betting that your fiber-optic cables can move memory faster than your GPU can do math.

Common Misconceptions

Misconception: Disaggregation means splitting the model layers (Pipeline Parallelism).

Reality: Pipeline parallelism slices the model horizontally (Layers 1-10 on GPU A, Layers 11-20 on GPU B). Prefill-Decode disaggregation slices the timeline of the request. Both the prefill GPU and the decode GPU hold a complete copy of the model weights; they just perform different phases of the timeline.

Misconception: The prefill GPU deletes its memory after sending the KV Cache.

Reality: In advanced systems (like PPD – Prefill Prefill-capable Decode), nodes heavily cache the KV data. If a user asks a follow-up question, the system can perform an “Append-Prefill” using the locally cached data, bypassing the need to resend the entire conversation history over the network.

Misconception: You can use standard ethernet to connect the clusters.

Reality: Attempting to stream gigabytes of KV Cache via standard TCP/IP introduces horrific latency. Disaggregation mathematically requires RDMA-capable networking (RoCE v2 or InfiniBand) to achieve zero-copy memory transfers.

What Most People Miss

The disruptive capability of Load-Aware KV Prefix Caching.

When analysts discuss KV transfer times, they assume the entire cache must be moved every time. What they miss is the integration of global prefix caching (like RadixAttention).

If 1,000 users ask a question about the exact same company PDF, a smart disaggregated router does not prefill the PDF 1,000 times. The prefill cluster calculates the KV Cache for the PDF once, blasts it via RDMA to the decode cluster, and pins it in the VRAM. For the next 999 users, the prefill cluster only calculates the math for their specific, 10-word question. It sends that microscopic sliver of math to the decode node, which instantly snaps it onto the giant, pre-loaded PDF cache. This effectively reduces the network transfer tax to near-zero for system-prompt-heavy enterprise applications.

Comparison Table

Case Study

Situation: In 2024, an enterprise AI platform serving thousands of concurrent coding assistants ran a standard monolithic vLLM cluster. As developers began pasting entire 50,000-line code repositories into the prompt box, the system began to collapse. Whenever a massive codebase was submitted, the prefill math completely monopolized the GPU, causing the autocomplete features for dozens of other developers on that same GPU to freeze for 2 to 3 seconds. The resulting tail-latency violated their core SLAs.

Challenge: Eliminate the decode stuttering caused by massive context prefilling, without purchasing exponentially more top-tier H100 GPUs.

Solution (The Splitwise Integration): The MLOps engineering team re-architected their serving infrastructure based on the Splitwise and DistServe frameworks. They split their hardware into two distinct pools. They routed all incoming prompts to a “Prefill Pool” of older, compute-heavy GPUs. They configured the prefill nodes to utilize NVIDIA NIXL libraries to execute zero-copy RDMA transfers of the resulting KV Caches over their existing RoCE v2 network, delivering them directly to a “Decode Pool” populated with memory-bandwidth-optimized silicon.

Outcome: The results fundamentally altered their unit economics. By physically isolating the workloads, the Decode Pool achieved a near-perfect, uninterrupted autoregressive generation rhythm. The massive code repositories were digested by the Prefill Pool without ever touching the decode GPUs. The platform achieved a 2.1x speedup in Time-to-First-Token (TTFT) and increased their overall system goodput by 1.8x, absorbing the massive agentic context windows without a single degradation in autocomplete fluidity.

Lessons Learned: The deployment validated that at scale, inference is a routing problem, not a raw compute problem. Forcing a GPU context-switch between prefill and decode destroys efficiency. Disaggregation proves that the network can effectively act as the backplane of a virtual, distributed AI processor.

Future Outlook

Next 12–24 Months

The era of Automated Dynamic Reprovisioning. Currently, cluster architects manually assign “Node A” to prefill and “Node B” to decode. Over the next year, frameworks like vLLM and SGLang will fully automate this. The central orchestrator will monitor real-time traffic. If a sudden surge of long-context documents hits the server, the orchestrator will instantly re-assign decode nodes to act as prefill nodes on the fly, dynamically morphing the physical cluster architecture second-by-second to match the mathematical shape of the incoming user demand.

Next 3–5 Years

The scaling of Disaggregated Memory Pools (CXL). The ultimate endgame of disaggregation is moving the KV Cache off the GPU entirely. Compute Express Link (CXL) 3.0 technologies will enable “Memory Pooling.” Instead of using RDMA to copy the cache from the Prefill GPU to the Decode GPU, both GPUs will simply plug into a massive, external rack of shared CXL memory. The Prefill GPU will write the cache to the shared rack, and the Decode GPU will instantly read it from the exact same rack, entirely eradicating the latency and bandwidth tax of network data transfers.

Next 10 Years

The Hardware-Native Disaggregated Silicon. By the mid-2030s, the concept of a “general purpose” AI accelerator will vanish in hyper-scale environments. Semiconductor manufacturers will design distinct physical server blades: a “Prefill Blade” packed entirely with dense matrix-multiplication ALUs and zero high-bandwidth memory, and a “Decode Blade” composed almost entirely of SRAM and HBM with minimal math cores. This absolute silicon specialization will push the cost of generating a token to fractions of a micro-cent, permanently commoditizing base-level intelligence.

Most Likely Scenario

As models continue to scale and agentic workflows demand million-token context windows, monolithic inference is mathematically doomed. The memory bandwidth required to generate tokens and the compute required to read prompts cannot be reconciled on the same chip at scale. Prefill-decode disaggregation, supported by blistering RDMA networking, has unequivocally won the architectural war, cementing the future of AI inference as a fundamentally distributed, highly specialized pipeline.

Key Takeaways

• When an AI answers a prompt, it does two things: it “reads” the prompt all at once (Prefill) and it “types” the answer one word at a time (Decode).
• Reading requires massive math power. Typing requires massive memory speed. Doing both on the same microchip wastes up to 80% of the hardware’s potential.
• Disaggregation physically splits the AI in half. One server cluster only reads prompts; another server cluster only types answers.
• This stops long, complicated prompts from freezing the system and interrupting the typing speed of other users on the network.
• To make this work, the two clusters must be connected by incredibly fast fiber-optic networks (RDMA) that beam the memory of the prompt directly from one GPU to the other in milliseconds.

Glossary

Compute-Bound: A task limited by the speed of the math processors. The Prefill phase is compute-bound because it processes thousands of tokens simultaneously.

Compute Express Link (CXL): A next-generation technology that allows multiple processors to seamlessly share a massive, external pool of memory.

Key-Value (KV) Cache: A massive, temporary memory file that the AI creates while reading the prompt. It acts as the AI’s “short-term memory” so it doesn’t have to re-read the prompt for every single word it generates.

Memory-Bandwidth-Bound: A task limited by how fast data can be physically moved from RAM into the processor. The Decode phase is memory-bound because it constantly waits for data to generate the next single word.

Remote Direct Memory Access (RDMA): A networking technology that allows one computer to read or write data directly to the memory (VRAM) of another computer, completely bypassing the operating system to achieve sub-millisecond speeds.

Time-to-First-Token (TTFT): The critical latency metric defining how long a user waits after hitting “Enter” before the AI types the very first word of its response.

Sources

ISCA 2024: Splitwise: Efficient Generative LLM Inference Using Phase Disaggregation

OSDI 2024: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

arXiv (2026): Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM

NVIDIA Technical Blog: Dynamo Disaggregation: Separating Prefill and Decode

Perplexity Research: Disaggregated Prefill and Decode architectures