Module 7 · Scheduling & placement

Prefill-Decode Scheduling and Disaggregation

Prefill and decode are two completely different workloads sharing one GPU. This page animates why they fight when co-located, how chunked prefill tames the interference, and how prefill-decode disaggregation splits them onto separate worker pools — shipping the KV cache between them.

Two phases, two bottlenecks

step-through

Prefill is compute-bound — one big parallel pass that saturates the math units. Decode is memory-bandwidth-bound — one token at a time, dominated by re-reading weights and the KV cache.

prefill work (compute-bound) decode work (bandwidth-bound) KV cache token

For a single request, prefill must finish before decode can begin. But across many requests a scheduler is free to choose the order. Prioritizing prefill or mixing prefill and decode in one batch raises throughput — at the cost of latency, because the in-flight decodes get slowed or stalled. The next figure shows that collision, and the fix.

The interference problem → chunked prefills

step-through

In a shared continuous batch a long prefill hogs an iteration and stalls the running decodes, spiking inter-token latency (TBT). Chunked prefill (Sarathi) splits that prefill into smaller pieces interleaved with decode steps, bounding the delay.

prefill iteration decode step (one token) stalled decode / latency spike

Tradeoff Chunking isn't free. Prefill is naturally one parallel pass; cutting it into pieces operates on smaller matrices (harder to saturate the GPU) and re-fetches weights and re-touches the KV cache each chunk (more bandwidth pressure). Make the chunks too small and prefill itself turns memory-bound — throughput falls. The chunk size is tuned to the largest token count the GPU can run while still hitting its SLO, given the decodes already in the batch.

Prefill-decode disaggregation: the KV handoff

step-through

DistServe goes further: run prefill and decode on separate worker pools. A prefill pool computes the prompt's KV cache, ships it over the interconnect to a decode pool, which streams out tokens. No more interference, and each pool can use its own hardware and parallelism.

prefill pool (compute) KV cache in transit decode pool (bandwidth) streamed token

Most systems co-locate the two phases so that one copy of the model weights and the KV cache can be reused in precious HBM. Disaggregation flips that: it replicates the weights across pools and moves the KV cache instead. That sounds expensive, but moving a request's KV — within a node or over a low-latency interconnect — is far faster than prefill or decode, so request latency barely moves (DistServe reports ~0.1% impact, p95 < 30 ms). DistServe pulls the KV from the prefill node's memory; Splitwise overlaps transfer with prefill by shipping each layer's KV as it's computed.

The tradeoff: isolation & scaling vs. KV bandwidth

auto-play

Each pool scales independently to match demand, and can sit on the hardware best suited to it (decode = lots of memory, little compute). The cost is a steady stream of KV cache crossing the interconnect between them.

prefill instances KV transfer (the cost) decode instances

Disaggregation buys clean isolation, simpler scheduling (no prefill-vs-decode prioritization to balance), better tail latency, and the freedom to scale and place each pool independently — at the price of replicated weights and KV cache moving over the network. It is one of several promising disaggregation targets; even attention vs. the feed-forward network have different enough profiles to be split in future.