Module 6 · Latency & throughput

Speculative Decoding

Decoding is memory-bound: each token costs one full sweep of the model's weights out of HBM. Speculative decoding amortizes that sweep over several tokens — a small fast model guesses ahead, the big model verifies the whole guess in one pass, and the output is identical to plain sampling.

One slow pass per token — versus many tokens per pass

step-through

Baseline decoding emits one token per expensive target forward pass. Speculative decoding drafts γ candidates cheaply, then verifies all of them in a single target pass.

token draft model (small) target model (large) accepted rejected / corrected

The win comes from amortization. Fetching the weights of a large model from HBM dominates the cost of a decode step, and that cost is roughly the same whether the target processes one token or a short batch of γ+1 tokens in parallel — much like prefill. So if the draft guesses well, several tokens "ride along" on a single weight sweep. Crucially, the accept/reject rule is designed so the final stream is distributionally identical to sampling from the target alone.

Acceptance dynamics: speedup rides on the acceptance rate

auto-play

Each round drafts γ tokens and verifies them in one target pass. Sometimes most are accepted (big speedup); sometimes the draft is rejected early (little gain). Verification cost ≈ one pass regardless of γ.

drafted accepted (kept) rejected + corrected one target pass

Tradeoff The lookahead window γ (commonly 3–12) is tuned empirically. Too small and you leave speedup on the table; too large and you waste compute drafting tokens that get thrown away. Speculative decoding trades extra compute for better memory-bandwidth use, so it shines most for large models at small batch sizes (memory-bound) and helps less as batches grow compute-bound.

Tree drafting — Medusa heads + tree attention

step-through

Instead of one linear guess, Medusa adds several decoding heads to the frozen target. Their top-k predictions form a tree of candidate continuations, all verified in a single pass via tree attention; the longest validated path is kept.

Medusa head / candidate frozen target trunk validated path (kept) pruned / rejected

Medusa's heads are independent — each predicts its position without seeing earlier heads' choices, so an early miss can invalidate a whole linear draft. The tree hedges against this: by validating several candidate sequences (the pruned Cartesian product of per-head top-k, the Medusa tree) in one pass, it keeps the candidate with the most accepted tokens.

Better drafters: EAGLE features & Multi-Token Prediction

step-through

All four schemes are just different ways to produce drafts. They trade off draft accuracy against memory and where the prediction happens — on tokens, on hidden features, or inside the model itself.

target trunk / features drafter (model / decoder / heads) draft tokens

EAGLE drafts at the feature level: a lightweight decoder reads the target's own hidden states instead of just its emitted tokens, cutting prediction uncertainty and lifting the acceptance rate. MTP bakes drafting into training — the model learns to predict the next n tokens via shallow per-token heads on a shared trunk, so any MTP model can spec-decode for free. The recurring theme: a cheaper, more accurate drafter means more tokens accepted per target pass.