2

Models

Read a model as a sequence of operations, representations, and memory transfers; its architecture predicts which inference resource will become scarce.

12 min read

Printed pages: 39–70

Star on GitHub

In one breath

Generative models compose large neural networks, and inference performance follows the work those networks demand. Language models tokenize a prompt, process every input position during prefill, store reusable attention state in a KV cache, and then decode one token per forward pass. Prefill is usually compute bound and largely determines time to first token (TTFT); decode is usually memory bound and determines tokens per second (TPS). Diffusion image and video systems instead refine an entire latent representation repeatedly and are generally compute bound. Arithmetic intensity, the ratio of operations to memory traffic, places each workload on a hardware roofline and explains why one optimization cannot accelerate every phase. Architecture knowledge turns performance tuning from folklore into bottleneck-directed engineering.

Architecture creates two different serving bottlenecks

Token input flows into Transformer work and then branches: Prefill path moves toward the Compute-bound region above the Roofline knee, while Decode path moves toward the Memory-bound region below the knee.

  • Token inputThe template and tokenizer produce the sequence processed by the model.
  • Transformer workAttention and feed-forward layers apply the learned weights.
  • Prefill pathMany input positions are processed in parallel while the KV cache is built.
  • Decode pathOne new token is produced per forward pass while weights are repeatedly read.
  • Roofline kneeHardware ops-to-byte balance separates bandwidth-limited from compute-limited work.
  • Compute-bound regionHigh arithmetic intensity makes available compute the limiting resource.
  • Memory-bound regionLow arithmetic intensity makes memory traffic the limiting resource.
  • Token inputTransformer work: enters model
  • Transformer workPrefill path: processes prompt
  • Transformer workDecode path: generates next token
  • Prefill pathCompute-bound region: high intensity
  • Decode pathMemory-bound region: low intensity
  • Roofline kneeCompute-bound region: above balance point
  • Roofline kneeMemory-bound region: below balance point

Why it matters

A model is not a uniform block of floating-point work. Linear layers hold much of a language model’s weight, attention moves and combines sequence state, sparse experts alter which parameters are active, and media pipelines call several component models over many denoising steps. These choices determine weight reads, intermediate tensors, cache growth, parallelism opportunities, and quality-sensitive approximations. An optimization aimed at the wrong resource can leave the expensive path unchanged.

The prefill and decode split is especially important because one request changes bottleneck midway through execution. Faster matrix computation helps a compute-heavy prompt phase, while memory movement and reuse dominate one-token-at-a-time generation. Treating both as one average hides the reason TTFT and TPS respond differently to batching, hardware, kernels, and placement. A useful model of inference must preserve phases rather than flattening them.

Reason from execution shape before model or accelerator names. A configuration file reveals layer count, hidden dimensions, attention heads, vocabulary, and whether feed-forward layers are dense or expert-routed. Request data reveals sequence length, batch, output length, and media dimensions. Together they predict the large tensors and repeated loops. Hardware then supplies compute and bandwidth ceilings; the implementation determines how closely work approaches them. This order survives product generations. It also explains why two variants of one architecture can inherit the same runtime support while still producing different capacity needs at different sizes, and why a newer GPU does not remove an algorithmic pattern that scales badly with context or latent volume. Starting from brand or peak specifications can assign supposedly faster hardware to decode that is actually limited by memory traffic, or apply token-generation techniques to whole-latent denoising. Starting from execution shape instead produces resource hypotheses to test before selecting the model, kernel, and hardware most likely to change the bottleneck.

Mental model

  • Neural networks transform representations through an input layer, hidden layers, and an output layer. Text encoders often expand chunks into high-dimensional semantic vectors, while image models compress millions of pixels into a smaller latent space. Encoders create representations; decoders use them to generate outputs; complete models and pipelines can compose several networks.
  • A linear layer multiplies an input vector by a learned weight matrix and adds a bias. Stacking only linear operations would collapse algebraically into one transformation, so activation functions insert nonlinearity between layers. This pairing of matrix multiplication and activation allows depth to carry useful internal structure.
  • A decoder-only language model maps tokens to embeddings, sends hidden states through many transformer blocks, and uses a language-modeling head to produce one logit per vocabulary item. Each block combines attention, a feed-forward network, normalization, residual paths, and activation. The feed-forward matrices hold most weights; attention supplies sequence relationships.
  • Two generation styles organize the chapter. Autoregressive models extend a token sequence one choice at a time. Diffusion models begin from noise and iteratively update a whole latent object. Both use transformers and attention, but their loop shape creates very different concurrency, memory, and latency behavior.
  • The roofline connects software work to hardware limits. Compare an operation’s arithmetic intensity with the GPU’s operations-per-byte balance point. Work below that point cannot feed compute fast enough and is memory bound; work above it consumes enough operations per byte to become compute bound.

Use a four-link chain when reading performance. First identify the semantic operation, such as attention, a feed-forward layer, expert routing, or denoising. Second identify execution shape: full-sequence matrix work, one-token vector work, sparse routing, or whole-latent updates. Third identify the stressed resource through bytes, operations, cache state, and repetition count. Finally connect that resource to the product metric: first-output latency, token cadence, image completion, video completion, capacity, or cost. Skipping a link encourages cargo-cult tuning: copying a kernel without checking local matrix shapes and data flow, or buying higher peak compute without confirming that memory is not the limit. Following the chain makes each proposed change testable and clarifies when a technique transfers across modalities or fails because the execution loop is fundamentally different. It also aligns communication: model researchers describe operations, systems engineers describe resources, and product teams describe visible outcomes against the same request and measurement. Remeasuring with identical inputs and workload slices confirms that an apparent gain did not come from changed data or test conditions.

Core ideas

  • Tokens are numeric identifiers for subword text fragments. A tokenizer and model-specific chat template flatten prompts, roles, tools, and other inputs into one sequence before neural computation starts. More efficient tokenization reduces sequence length and can improve end-to-end inference because fewer positions must be processed or generated.
  • Prefill processes the full input sequence, computes attention, and creates the KV cache. Decode repeatedly performs a forward pass, converts vocabulary logits into probabilities, selects a token under temperature or top-k and top-p constraints, updates state, and stops at a stop token or length limit.
  • Attention compares a query with keys and uses the resulting scores to combine values. Causal self-attention relates a language token only to allowed positions in its sequence; cross-attention conditions one sequence or modality on another. Multiple heads can learn different relationship patterns in parallel.
  • Without reuse, each new token would recompute keys and values for prior positions. The KV cache stores those results, turning incremental decode attention from repeated quadratic work into work that grows linearly with existing sequence length. It is built in prefill, read and extended in decode, and normally occupies GPU memory.
  • A mixture of experts (MoE) replaces one dense feed-forward matrix with many expert matrices plus a router that activates a small subset for each token at each layer. Active parameters can be far fewer than total parameters for one request, although batches may collectively touch most experts. The structure enables expert-oriented multi-GPU parallelism.
  • Image generation is a pipeline rather than one monolith: a text encoder represents the prompt, a denoiser repeatedly updates noisy latent data under conditioning, and a variational autoencoder converts the final latent to pixels. Working in compressed latent space makes whole-image attention feasible.
  • A conventional image request often uses thirty to fifty denoising steps, with conditioned and unconditioned forward passes combined by guidance. Resolution, step count, guidance, prompts, and negative prompts all affect work or output. Few-step models reduce the loop to eight or fewer steps but accept a quality trade-off.
  • Modern video models keep space and time together in one latent volume so every frame can attend to others during each update, avoiding frame-by-frame error accumulation. The price is enormous compute over fixed, short clips; the book’s snapshot describes models that often devote an eight-GPU node to one request.

How it works

  1. Apply the model’s chat template and tokenizer. Count input, optional reasoning, and maximum output tokens against the context window, because sequence length affects attention work, cache size, and total response time.
  2. Run prefill over the input matrix. Large matrix multiplications reuse weights across many positions, produce high arithmetic intensity, and create attention keys and values for every prompt token. This compute-heavy phase sets the first-output wait.
  3. Run the transformer once per output token. Each pass reads model weights for relatively small vector-matrix work, produces vocabulary logits, applies selection rules, and appends the chosen token plus new cache state. Low arithmetic intensity makes memory movement the usual limit.
  4. Inside a block, form query, key, and value representations, calculate attention scores, normalize them, and combine values. Then pass the hidden state through feed-forward layers and residual normalization paths. This repeats across many blocks before the output head.
  5. For images, encode the text, initialize a latent with noise, and update the whole latent across repeated denoising steps. Combine conditioned and unconditioned passes using the guidance setting, then decode the finished latent into pixels.
  6. For video, extend the latent with a time dimension and update all frames together. Attention can preserve global consistency better than a framewise chain, but the much larger latent and model turn each denoising step into an expensive compute task.
  7. For the expensive kernel, count floating-point operations and bytes read or written. Divide work by traffic, compare the result with hardware operations per byte, and optimize the saturated resource. Recalculate when sequence shape, batch size, precision, implementation, model, or hardware changes.
Arithmetic intensity = floating-point operations ÷ bytes moved
The ratio describes one operation’s reuse of loaded data; comparing it with the hardware balance point identifies a memory-bound or compute-bound region.

Metrics that matter

Arithmetic intensity

Operations per byte moved

Locates an operation relative to the hardware roofline knee.

Hardware balance

Peak operations per second ÷ peak bytes per second

Provides the comparison point between memory and compute limits for a chosen precision.

Prefill latency

Primary contributor to TTFT

Tracks the compute-heavy work of processing prompt positions and constructing the KV cache.

Decode cadence

Primary contributor to TPS

Tracks repeated weight reads and one-token generation in the memory-heavy phase.

KV-cache footprint

State grows with active sequence length

Signals memory capacity and movement pressure created by retained attention state.

Denoising work

Steps × forward passes per step

Connects image or video latency to loop length, guidance, latent size, and model cost.

Trade-offs

Implementation versus algorithm

FlashAttention-style fused kernels remove excess memory traffic without changing model outputs, and PagedAttention manages fragmented KV-cache blocks; both preserve the underlying attention rule and scaling. Sliding-window, gated, linear, compressed, or multi-latent attention changes the work and may trade quality for better time or space complexity. State-space designs such as Mamba replace attention with recurrent state updates, while hybrid models mix both families.

Dense versus sparse experts

Dense models use every parameter predictably. MoE reduces active parameters for an individual token, but routing, expert placement, and diverse batched requests complicate memory use and multi-GPU execution.

Recomputation versus cache memory

The KV cache avoids repeating prior key and value calculations, turning decode attention into practical incremental work. It consumes increasing GPU memory and motivates paging, placement, and reuse strategies.

Denoising steps versus image quality

More diffusion steps provide repeated refinement but add forward passes. Few-step consistency or distilled models can be dramatically faster, with visibly lower quality that may still suit latency-sensitive effects.

Global video consistency versus compute

Updating the full space-time latent limits framewise drift and lets frames attend across time, but fixes short output lengths and requires very expensive attention. Autoregressive elements aim to relax that cost without returning to uncontrolled error accumulation.

Single request versus batching

Decode for one request has low arithmetic intensity. Batching uses the same weight load for more work and moves the operation toward compute use, but changes latency, cache demand, and which experts become active.

Engineering checklist

  • Inspect model configuration, layer dimensions, attention form, density, and component pipeline before choosing a runtime path.
  • Use the correct tokenizer and chat template, and record input, reasoning, and output token lengths.
  • Profile prefill and decode separately so TTFT and TPS are not collapsed into one average.
  • Estimate operations and memory traffic for the expensive kernel before optimizing compute or bandwidth.
  • Account for KV-cache growth, fragmentation, access, and lifecycle across active sequences.
  • Measure expert activation under real batches rather than assuming active parameters equal total server work.
  • For media, record latent dimensions, denoising steps, guidance passes, component models, output size, and batch size.
  • Classify an attention optimization as lossless implementation work or a quality-affecting algorithm change and validate accordingly.

Vocabulary

Hidden state
An intermediate vector representation passed between neural-network layers.
Matrix multiplication
The core operation that applies learned weight matrices to input vectors or matrices.
Prefill
The phase that processes all input positions and constructs the request’s attention cache.
Decode
The autoregressive phase that repeatedly produces and selects one next token.
KV cache
Stored attention keys and values for prior tokens, reused and extended during decode.
MoE
A sparse architecture whose router selects a subset of expert matrices for each token and layer.
Latent space
A compressed internal representation in which media models perform generation work.
Arithmetic intensity
The ratio of computation performed to bytes transferred for an operation.
Roofline model
A chart that relates arithmetic intensity to bandwidth and peak-compute performance ceilings.
Paged attention
An implementation that stores cache blocks through a lookup table rather than one contiguous allocation.

Source map

  • Printed pages 41–42

    Introduces neural models, transformers, autoregression, and iterative denoising.

  • Printed pages 42–46

    Builds intuition for representations, layers, matmul, and nonlinear activation.

  • Printed pages 46–49

    Traces token preparation, prefill, decode, logits, sampling, and stopping.

  • Printed pages 49–53

    Explains transformer blocks, attention inputs, heads, masks, and KV caching.

  • Printed pages 53–54

    Describes sparse expert routing and its production batching behavior.

  • Printed pages 55–59

    Maps image pipelines, latent diffusion, guidance, architecture growth, and few-step models.

  • Printed pages 59–61

    Explains space-time latent generation, global attention, and resource limits.

  • Printed pages 61–63

    Defines arithmetic intensity and the memory-to-compute roofline transition.

  • Printed pages 63–67

    Derives prefill, decode, image, and video bottlenecks from work and memory traffic.

  • Printed pages 67–70

    Surveys implementation, paging, approximate attention, and state-space approaches.

Generative-model foundations, transformers, autoregressive generation, and iterative denoising
Printed pages: 41–42; PDF pages: 43–44
Neural-network layers, representations, encoders, decoders, matmul, and activation functions
Printed pages: 42–46; PDF pages: 44–48
Tokens, templates, sequences, prefill, decode, logits, sampling, and stop conditions
Printed pages: 46–49; PDF pages: 48–51
Architecture metadata, transformer blocks, feed-forward layers, attention, and KV caching
Printed pages: 49–53; PDF pages: 51–55
Mixture-of-experts routing, active parameters, batching effects, and expert parallelism
Printed pages: 53–54; PDF pages: 55–56
Image-generation pipelines, latent diffusion, guidance, architectures, and few-step models
Printed pages: 55–59; PDF pages: 57–61
Video latent space, temporal attention, fixed output length, compute cost, and autoregressive directions
Printed pages: 59–61; PDF pages: 61–63
Compute, memory bandwidth, ops-to-byte balance, arithmetic intensity, and roofline regions
Printed pages: 61–63; PDF pages: 63–65
Prefill, decode, media bottlenecks, and the memory movement behind attention
Printed pages: 63–67; PDF pages: 65–69
Attention implementation, paging, approximate variants, windowing, compression, and state-space alternatives
Printed pages: 67–70; PDF pages: 69–72