Inference Runtimes: the serving stacks
A model isn't one giant GPU kernel — it's a graph of kernels (matmul, ReLU, attention) composed and driven by a runtime. This finale tours the production stacks that do that driving — TensorRT‑LLM, Triton, vLLM, and SGLang — and shows how every technique from the series lives inside them.
The serving stack: a request's round trip
step-throughFollow one request down the stack — gateway, scheduler, engine, kernels, GPU — and watch generated tokens stream back up.
That whole column is what a runtime provides. The article splits the field two ways.
NVIDIA's stack keeps a clean frontend / backend divide — TensorRT(-LLM) is the
backend that runs the model on GPUs, and the Triton Inference Server is the frontend that
routes and batches requests to backends. The cross-platform stacks — vLLM and
SGLang — flatten that divide, providing the network face, the scheduler, and the
kernels all in one.
Four runtimes, four signatures
auto-playEach card pulses its defining trait. They map straight back to earlier modules — paging, prefix caching, kernel fusion, and orchestration.
Inside the engine, at work
auto-playThe whole series in one frame: the scheduler forms a continuous batch, KV lives in paged blocks, a new request hits a prefix-cache, and tokens stream out.
One last distinction the article draws: when the optimization happens.
TensorRT does it ahead of time in a one-time compile — analyse the graph, pick kernels, fuse
ops, quantize for the target GPU. vLLM and SGLang have no explicit compile step; they optimize a
little at startup (a kernel pipeline they cache and reuse, optionally via
torch.compile) and spend the rest of their effort on runtime scheduling. Same goal,
different split between static and dynamic.
Static vs dynamic: where the optimization lives
step-throughA model's journey to served tokens crosses two regimes — an offline compile phase and an online scheduling phase. Engines sit at different points along that line.