Techniques

Optimization is a diagnosis problem. Quantization reduces compute and memory work but spends a quality budget. Speculation uses spare decode compute to improve inter-token latency (ITL) and tokens per second (TPS). Caching avoids repeated prefill and improves time to first token (TTFT). Parallelism makes models fit or changes latency and throughput across devices. Disaggregation separates compute-bound prefill from memory-bound decode at sufficient scale. Measure the bottleneck, apply the narrowest suitable lever, and retest the whole system because techniques can reinforce or obstruct one another.

Think in four budgets: compute, memory capacity and bandwidth, communication, and acceptable quality change. Each technique moves pressure between budgets. Quantization stores and moves fewer bits while risking numerical error; speculation spends compute to reduce decode iterations; caching spends memory to avoid prefill; parallelism spends communication to gain capacity or concurrency; disaggregation spends hardware and transfer bandwidth to specialize phases. A good configuration removes the dominant constraint without making the receiving budget dominant.

Suppose measurement shows compute-bound prefill, bandwidth-bound decode, and too little key-value cache headroom after weights and buffers. A precision change can reduce matrix cost, bytes moved, and cache footprint, but do not lower the whole model at once. First convert less-sensitive linear weights and compare perplexity, public benchmarks, and product evaluations with the original model; then add activations; consider key-value state only when long context or disaggregation is actually limited by cache capacity or transfer. Keep input-output layers, outlier-heavy regions, and attention math at higher precision when needed so recurrent errors do not compound token by token. At each stage measure time to first token, tokens per second, memory, throughput, and quality. The goal is not the fewest bits, but the lowest latency and cost while the quality difference remains within repeated-run noise.

For an interactive coding workload, put the stable system prompt, tool definitions, and repository context before the current edit and new question so prefix caching can skip substantial prefill. Route follow-up requests for the same user or codebase toward a replica with the hot cache, but cap affinity so one replica does not overload. During low-batch decode, repetitive code may yield long accepted n-gram drafts and fewer target-model forward passes. As concurrency and batch consume spare compute, shorten or disable speculation while prefix reuse still saves prefill. Track skipped input tokens, cache hits, draft acceptance, accepted tokens per iteration, and net inter-token latency separately; the techniques act on different phases and should not share one on-off switch or be judged only by total throughput.

When a model exceeds one GPU, include weights, runtime workspace, activations, target batch, and longest-context cache headroom before choosing a parallel label. Tensor parallelism (TP) splits tensors within every layer so GPUs share weight reads and matrix multiplication; fast intra-node links can lower per-user latency, but each layer synchronizes and scaling reverses when communication exceeds saved work. Expert parallelism (EP) places intact experts on devices and routes tokens without a TP-style all-reduce after every layer, so mixture of experts (MoE) throughput can scale farther, including across nodes. Pipeline parallelism (PP) assigns intact layer stages across nodes, introducing sequential bubbles; it is mainly a capacity solution when a dense model cannot otherwise fit. Prove each added device with parallel efficiency and interconnect traces. If the model fits comfortably in one node, another replica is often simpler and serves more independent requests.

Disaggregation becomes plausible when a large, heavily used model receives enough long uncached inputs for compute-bound prefill to disturb memory-bound decode. Send each request first to a decode-side cache check: handle full hits or short prefill locally, and send only long misses to the prefill pool. Prefill workers can use compute-oriented GPU count, TP, and batch settings; after producing the first token and key-value blocks, they transfer state over the chosen interconnect to decode workers tuned for bandwidth, cache capacity, and subsequent tokens. The pool ratio is an operating variable, not fixed one-to-one. A growing prefill queue asks for more front-end capacity; high decode occupancy or eviction asks for memory; slow transfer may require cache quantization, nearer topology, or a higher local-prefill threshold; light traffic may warrant collapsing the split to avoid idle GPUs.

Roll out one lever behind a controlled traffic slice and retain the unoptimized path plus fast rollback. Segment results by input and output length, cache hit or miss, batch, concurrency, hardware, and topology; a mean can hide long-context regressions or mistake a low-load gain for peak capacity. Gates should cover percentile latency, system throughput, GPU memory, unit cost, errors, and quality. Quantization also needs product evaluation, speculation needs acceptance across temperatures, and caching needs tokens skipped rather than hit rate alone. Observe a full peak cycle before increasing traffic. Keep triggers that disable the technique when its premise disappears: high batch stops speculation, a quality breach restores original precision, and an overloaded hot replica reduces cache affinity.

Treat the prefill-to-decode worker ratio as a workload-dependent setting. Long inputs and prefill-heavy traffic need more prefill capacity, while decode-heavy traffic needs more decode capacity. Monitor the prefill queue and decode key-value cache, and change the ratio at runtime as traffic changes instead of fixing one prefill engine to one decode engine. For short inputs or cache hits, conditional disaggregation can keep prefill on the decode engine. At lower traffic, horizontal replicas can be more efficient than disaggregation, whose extra GPUs, queue, and cache transfer add cost.

Document the causal chain for every lever: the measured phase and resource bottleneck, enabling traffic condition, budget receiving pressure, expected user or cost benefit, new failure mode, quality gate, disable threshold, and rollback. For example, 'low-batch decode has spare compute, so enable short-draft speculation and disable it when acceptance or net ITL falls below the gate' is more operable than 'enable EAGLE.' This record turns a configuration into an observable policy and makes the revalidation scope visible when another optimization changes the same compute, memory, or communication budget.