Inference Glossary

A filterable decision vocabulary for models, systems, optimization, and production.

6 min read

Printed pages: 209–230

reference
terminology

Filter glossary terms

34 terms

Use terms as diagnostic handles

Inference discussions become clearer when every metric names an observation, every bottleneck names a limiting resource, and every technique names the mechanism it changes. This condensed glossary favors terms that help an engineer move from a product symptom to a measurable hypothesis and then to an appropriate serving lever.

Measurement and workload

Time to first token: Time to first token, abbreviated TTFT, measures how long a user waits before the first generated token becomes available. It includes queueing and prefill work, so a high value is not automatically a decode problem.
Inter-token latency: Inter-token latency, abbreviated ITL, is the delay between successive streamed tokens during decode. Its reciprocal corresponds to the per-user generation rate when other client overhead is small.
Perceived tokens per second: Perceived tokens per second, abbreviated TPS, is the rate at which one user receives streamed output. It is approximately the reciprocal of inter-token latency when client overhead is small, and it differs from aggregate system throughput.
Throughput: Total useful work completed per unit time, such as aggregate tokens across all requests. It is a system-capacity measure and should not be confused with the generation speed perceived by one user.
Latency percentiles: A distribution view such as the median, 95th, or 99th percentile. Percentiles expose slow-tail behavior that an average hides and should be segmented by request shape and load.
Load testing: Sustained, controlled traffic used to reveal saturation, queue growth, autoscaling behavior, and failure limits. A useful test reproduces arrival timing and sequence shapes rather than only sending identical requests as fast as possible.

Model execution

Prefill: The phase that processes the whole input sequence and constructs attention state before generation begins. Large matrix operations often make it compute-limited, and its duration strongly influences first-response latency.
Decode: The autoregressive phase that repeatedly produces one next token for each active sequence. It rereads model weights and stored attention state, so memory movement frequently limits performance.
Continuous batching: Token-level scheduling that inserts and removes requests as individual sequences progress. It keeps accelerator slots busy without forcing every request to wait for an entire fixed batch to finish.
Key-value cache (KV cache): The key-value cache, usually called the KV cache, stores attention keys and values for tokens already processed. Reuse avoids recomputing prior context, but the cache consumes device memory and grows with sequence length and concurrent requests.
Prefix caching: Reuse of key-value state for an identical leading prompt across requests. It can skip much of prefill for repeated system prompts, conversations, and code contexts when routing preserves locality.

Bottlenecks and hardware

Arithmetic intensity: The amount of computation performed for each byte moved. Comparing it with a processor's compute-to-bandwidth balance helps predict whether faster arithmetic or less data movement is the more promising optimization.
Compute-bound: A workload whose rate is limited primarily by available arithmetic capacity. Increasing bandwidth alone will not remove the constraint; fewer operations, lower precision, or more effective compute use may.
Roofline model: A visual model that places an operation's arithmetic intensity against hardware bandwidth and compute ceilings. It turns a profile into a bottleneck hypothesis rather than a list of utilization percentages.
Video random-access memory: Video random-access memory, abbreviated VRAM, holds model weights, activations, and attention cache on an accelerator. Capacity limits what fits, while bandwidth often limits how quickly decode can stream tokens.
GPU interconnect: The communication path among accelerators, such as a high-bandwidth intra-node fabric or a slower inter-node network. Parallel strategies must fit its bandwidth and synchronization cost.

Runtime and kernels

Tensor Core: A specialized GPU arithmetic unit for mixed-precision matrix multiply-accumulate operations. Inference performance depends on selecting shapes and number formats that let kernels use these units effectively.
CUDA: NVIDIA's programming platform for GPU memory, kernels, graphs, and execution. It spans low-level drivers and developer-facing runtime facilities used by higher-level inference frameworks.
Kernel fusion: Combining neighboring operations into one GPU kernel so intermediate values remain close to compute instead of making extra trips through device memory. The gain depends on the original launch and memory overhead.
Inference engine: A serving runtime that schedules model execution and exposes optimizations such as batching, caching, quantization, and speculation. Engine choice should follow model support and measured workload behavior.
Profiler: A tool that attributes time and memory to host work, transfers, kernels, and synchronization. It is most useful after a reproducible benchmark establishes what user-facing regression needs explanation.

Optimization and scaling

Quantization: Representing weights, activations, or cache with fewer bits to reduce storage, bandwidth, and sometimes compute cost. Every deployment needs quality evaluation because sensitivity varies by model, tensor, and workload.
Tensor parallelism: Tensor parallelism, abbreviated TP, divides individual tensor operations across GPUs. It can improve single-request latency within a fast-connected node but introduces frequent collective communication.
Mixture of experts: A mixture of experts, abbreviated MoE, divides some linear-layer weights into sparse experts and routes each token through a selected subset. It expands model capacity without activating every expert on every forward pass.
Expert parallelism: Expert parallelism, abbreviated EP, distributes the experts of a mixture-of-experts model across GPUs. Tokens travel to selected experts, trading communication complexity for sparse model capacity and throughput.
Pipeline parallelism: Pipeline parallelism, abbreviated PP, places different layer stages on different GPUs. It can make a dense model fit across nodes, but pipeline bubbles leave some devices idle while stages wait on one another.
Speculative decoding: A family of methods that proposes and validates draft tokens so decode can accept multiple tokens from one target-model forward pass. Its benefit depends on draft acceptance and verification overhead.
Prefill-decode disaggregation: Running prefill and decode on separately scaled serving resources. Isolation can improve scheduling and hardware matching, but transferring attention state adds networking and operational costs.
Cache-aware routing: Sending a request to a replica that already holds a useful prompt prefix or adapter. Better locality can lower first-response delay, but it must be balanced against replica load and failover needs.

Production and modalities

Autoscaling: Automatically changing the number of serving replicas in response to demand or utilization. Its control window, startup delay, queue policy, and safety margin jointly determine latency and idle cost.
Cold start: The interval from requesting new capacity until it can return a successful result. Provisioning, image startup, weight loading, and runtime compilation can each dominate a different deployment.
Vision-language model: A vision language model, abbreviated VLM, accepts visual inputs together with text and returns text. Image preprocessing, visual tokens, and long context alter its workload compared with text-only generation.
Automatic speech recognition: Automatic speech recognition, abbreviated ASR, converts audio into text. A production pipeline may also detect speech regions and identify speakers; decoder work dominates runtime and can benefit from language-model-style batching.
Text to speech: Text to speech, abbreviated TTS, produces audio from text. In token-based systems, a performant audio decoder such as SNAC can be paired with the generated TTS token stream.

Source map

Foundations
Printed pages 211 to 214 cover application framing, arithmetic intensity, batching, autoscaling, and early model terminology.
Execution and deployment
Printed pages 215 to 218 cover CUDA, decode, denoising, deployment, and data-format concepts.
Memory and model behavior
Printed pages 219 to 222 cover accelerators, serving metrics, key-value state, model families, and multi-GPU execution.
Optimization and reliability
Printed pages 223 to 226 cover parallelism, caching, quantization, routing, service objectives, and speculation.
Serving and modalities
Printed pages 227 to 229 cover latency, throughput, runtimes, vision, audio, transports, and device memory.

Foundational application, model, batching, and performance terminology: Printed pages: 211–214; PDF pages: 213–216
CUDA, execution, model-format, and deployment terminology: Printed pages: 215–218; PDF pages: 217–220
GPU, memory, latency, caching, model, and multi-GPU terminology: Printed pages: 219–222; PDF pages: 221–224
Online serving, parallelism, quantization, routing, and reliability terminology: Printed pages: 223–226; PDF pages: 225–228
Serving metrics, runtimes, modalities, transports, and device-memory terminology: Printed pages: 227–229; PDF pages: 229–231