KV Cache Management and Offload
Real workloads aren't one-and-done. Requests share long prefixes — system prompts, few-shot examples, chat history. If we keep those KV blocks around, we can reuse them instead of recomputing, and tier colder blocks down a memory hierarchy when HBM runs out.
Prefix caching: compute the shared prefix once, reuse it
step-throughThree requests share a long system prompt. The prefix KV is computed on the first request and reused on the rest — only each request's unique suffix needs fresh prefill.
The KV cache normally trades compute for memory within one request, then is discarded. Prefix caching (a.k.a. prompt caching) lets parts of that cache outlive a request and be reused across later, non-concurrent requests. The win is largest when a big preamble — a long system prompt or a multi-turn history — is identical across many requests. But how do we know what's already cached, and how much of a new prompt matches? That's what the index below answers.
RadixAttention: indexing prefixes in a radix tree
step-throughCached prefixes live in a radix tree keyed by tokens. A new request walks the tree, reuses the longest matching prefix, and branches off for its new tokens. Cold branches are evicted by LRU.
LMCache and
CacheBlend relax this by hashing chunks and reusing partial matches.)
KV offload: tiering blocks across the memory hierarchy
step-throughHBM is scarce, so colder prefix blocks are offloaded down a hierarchy — GPU HBM → CPU DRAM → NVMe — and prefetched back up on a hit. Going down gains capacity but loses bandwidth.
Caching prefixes in HBM eats into the memory available for active request decoding, so as contexts
grow, something has to give. Systems like LMCache delegate offload to host DRAM, local NVMe,
or remote hosts. Two tricks make this fast: aggregating the paged-out KV into one big
contiguous buffer (instead of many tiny I/Os), and fetching layer by layer while overlapping I/O
with compute so the request doesn't stall. The loop below shows the decision that makes offload
worth it.
Putting it together: load the KV, don't recompute it
auto-playOn a prefix hit served from an offloaded tier, loading the cached KV beats re-running prefill. The two lanes race; loading from DRAM/NVMe wins when the prefix is long.