Prerequisites

Optimization is a search for the best workable balance, not a contest to maximize one number. Begin with the application’s model needs, input and output interface, end-to-end latency budget, unit economics, traffic pattern, and compliance boundary. Those constraints determine whether shared inference is still appropriate, which models deserve evaluation, and whether latency or throughput should dominate. For streamed language-model output, time to first token (TTFT) describes the wait until output begins and tokens per second (TPS) describes delivery after that point. Report their percentiles, distinguish per-user speed from service-wide throughput, and compare inference-only timing with what the client experiences. A technically faster model is not an improvement if it misses the product’s quality, cost, or reliability contract.

Inference improvements exchange latency, throughput, quality, and cost. Without a product definition, the team cannot tell which exchange is acceptable. A highly batched deployment may process an offline corpus efficiently yet feel unusable in chat. A premium low-latency service may delight a business workflow but destroy the margins of a volatile consumer product. Constraints reduce the search space and turn an attractive benchmark into a decision that can be evaluated against users, traffic, and economics.

The order of operations matters. Choosing a smaller capable model can change cost and latency more than a later engine optimization. Product-specific evaluations prevent engineers from accelerating a model that is not good enough, and they provide the baseline needed when an optimization might reduce quality. Only after these decisions are grounded should the team take on dedicated inference and its larger operational surface.

Constraints also need an order. Quality establishes whether the feature works at all. Interaction style establishes which part of latency users can perceive. Traffic and economics establish how much capacity the product can afford. Availability and compliance establish where and how the service may run. When two goals conflict, this hierarchy prevents a benchmark from silently redefining the product. It also makes an optimization proposal falsifiable: the team can state which constraint should improve, which one may worsen, how much movement is acceptable, and which evaluation or production measurement will decide. For example, chat may accept slightly lower total throughput for faster first output, while offline document processing may accept slower items if hourly completions rise. This is not a universal ranking; it requires each product to write its own. This discipline turns performance work from an open-ended search into a bounded engineering experiment.

Evaluation is the bridge between product language and serving language. A product owner can describe unacceptable failures and the hardest successful cases; the evaluation set turns them into repeatable evidence. An inference engineer can then compare a smaller model, a fine-tuned variant, or a quality-risking optimization without guessing whether the behavioral change matters. This bridge must remain attached to workload slices. A model that passes on short requests may fail on long context, and a service that looks economical at average traffic may miss the latency budget at concurrency peaks. If evaluation measures only capability, performance tests use synthetic inputs, and cost estimates assume another traffic pattern, those three sets of numbers cannot answer the same product question. The contract, evaluation, and serving measurements should therefore use matching scenarios, input distributions, and identifiers so every comparison traces to one use case.

A metric needs a protocol, not only a name. Record where the timer begins and ends, whether output is streamed, the prompt and output lengths, concurrency, serving mode, and the percentile being reported. For TPS, state whether the numerator covers one response or the entire service. Pair latency with the evaluation result and cost for the same workload, since a faster run with degraded answers or a different request mix is not comparable. Finally, align inference-only and end-to-end traces by request. When model time stays flat while client time rises, the evidence points toward queueing, network distance, or application work instead of another model optimization. The protocol should also fix load generation and warm-up state, avoiding cold starts, burst queues, and steady execution collapsed into one mean. Alongside central tendency, retain P90, P95, or P99 tails and the sample window. Then model, deployment, or version results become comparable and user complaints map to a specific service stage.