How LLM inference really works
An animated, click-through companion to IOP Systems' Inference Infrastructure series — eleven modules taking you from a single attention head all the way to a production serving stack. Every concept is a hands-on animation you can play, pause, and step through at your own pace.
LLMs & Transformers
Embeddings, attention (Q·Kᵀ→softmax→V), multi-head, MHA/MQA/GQA/MLA, and Mixture-of-Experts.
5 figures · step + autoInference & the KV Cache
Prefill vs decode, why recomputing K/V is wasteful, and how the cache trades memory for speed.
4 figures · step + autoSharding a Model
Pipeline parallelism & bubbles, tensor parallelism with all-reduce, and expert parallelism.
4 figures · step + autoBatching, Scheduling & Paging
Static vs continuous batching (Orca) and PagedAttention's block table for the KV cache.
4 figures · step + autoI/O-Aware Kernels
Why attention is memory-bound, and how FlashAttention tiles + online softmax avoid the k×k matrix.
4 figures · step + autoSpeculative Decoding
Draft-then-verify, parallel acceptance, plus EAGLE, Medusa trees and Multi-Token Prediction.
4 figures · step + autoPrefill-Decode Disaggregation
Chunked prefill to stop interference, and splitting prefill & decode into separate pools.
4 figures · step + autoKV Cache Management & Offload
Prefix caching with a radix tree, and offloading cold KV blocks down the memory hierarchy.
4 figures · step + autoOverview of Training
Pretraining → SFT → RLHF/RLAIF, reward models, LoRA, and quantization for cheaper inference.
4 figures · step + autoGPU Hardware
SMs & warps, the register→SRAM→HBM hierarchy, CUDA/ROCm, and writing tiled Triton kernels.
5 figures · step + autoInference Runtimes
The serving stack and how vLLM, SGLang, TensorRT-LLM and Triton package every technique above.
4 figures · step + auto