Inference

Training creates model weights; inference puts those weights to work for real requests. For classic machine learning, that production step could be modest. Generative models change the engineering problem: obtaining weights and GPUs does not by itself produce a service that is fast, economical, or dependable under load. The discipline therefore spans three cooperating layers. Runtime improves a model on one GPU-backed instance, infrastructure coordinates many instances across increasingly large resource pools, and tooling gives engineers a productive level of control. Specialization pays when a real product requirement cannot be met by generic serving alone. Before that threshold, extra machinery is merely operating burden; after it, local fixes at only one layer leave the system bottlenecked elsewhere.

A fast model server is not yet a production inference system. Its traffic can exceed the capacity of one instance; a regional outage can erase otherwise excellent kernel work; or an interface can hide the controls needed to diagnose a quality or latency regression. These are different failure modes, but users experience one product. Treating runtime, infrastructure, and tooling as a single system prevents a team from celebrating a local speedup while queueing, placement, capacity, or operator friction still dominates the result.

Inference engineering begins where merely serving a function ends. The trigger is not a fashionable optimization or a fixed GPU count. It is evidence that the workload needs deliberate control over performance, scale, availability, or developer experience. At small scale the problem may be one efficient deployment. With more traffic it becomes autoscaling, then regional and provider capacity, and eventually global scheduling over a unified pool. The boundary moves with the product, so the team should add specialization in response to demonstrated constraints rather than adopting the whole stack in advance.

The scale ladder is also a diagnostic tool. If one replica is wasteful, adding replicas multiplies waste; runtime work comes first. If replicas are efficient but traffic arrives unevenly, routing and autoscaling become the constraint. If enough GPUs exist in aggregate but sit behind regional or provider boundaries, capacity coordination becomes the problem. If operators cannot express these choices safely, the abstraction is now the limiting layer. This sequence keeps the team’s question concrete: what resource or control prevents the next unit of demand from being served within the product contract? The answer identifies where engineering effort belongs today, while leaving the other layers ready for the next change in scale. This does not mean problems always arrive in a fixed order; it requires remeasurement before each expansion. When usage patterns, the model, or regional distribution changes, a previously secondary layer can immediately become the main bottleneck. Keeping each layer’s responsibilities and signals clear lets the team shift priorities without replacing the whole system.

Useful layer boundaries are interfaces, not walls. Runtime should report capacity and failure signals that infrastructure can act on. Infrastructure should expose placement and load conditions that explain what users observe. Tooling should translate those facts into controls without erasing them. This model helps with ownership: a team may buy a managed layer, build it internally, or combine both, yet it still needs to know which guarantees cross each boundary. The book’s map is therefore a way to reason about a complete service even when the implementation comes from several vendors and internal teams. Missing responsibility at an interface is as consequential as a slow component inside one layer. If runtime supplies no trustworthy capacity data, autoscaling can only guess; if infrastructure does not explain where requests went, model latency is hard to interpret; if tooling exposes buttons without state, operators cannot build confidence. Explicit interface contracts let each layer evolve independently while the system stays coherent.

Read these signals together. Higher single-instance capacity is valuable only if the fleet can route work to it and keep it available. Spare capacity protects bursts only if new replicas become ready in time. Geographic diversity helps resilience only if traffic can fail over, and nearby capacity helps latency only if placement does not fragment the fleet into unusable silos. Operability is the final check: a technically capable stack that cannot be understood or changed safely will not remain reliable. No one metric represents inference engineering; the service contract determines which combination must stay inside bounds. Teams should also distinguish leading signals from lagging outcomes. Fleet headroom and replica readiness can warn of risk before users are affected; end-to-end latency and success rate describe the outcome users already feel; change time reveals whether tooling lets fixes land safely. Connecting all three traces symptoms to the correct layer instead of tuning the model whenever latency rises. Measurement windows should include routine traffic, peak load, and failover so calm-period numbers do not hide real risk.