Production

A fast graphics processing unit (GPU) process is not yet a production service. Package a reproducible runtime, understand its concurrency envelope, keep excess work in a bounded queue, scale replicas before latency fails, route each request to suitable state, and make cold starts short enough for the policy to react. Add capacity and failure domains deliberately, roll out with live evidence, secure data and weights, account for total cost of ownership (TCO), and observe the full request. Time to first token (TTFT) and tokens per second (TPS) matter, but client setup, network, queue, and protocol also consume the service-level agreement (SLA).

Start with an immutable serving artifact. A container image layers a proven base, exact application and system dependencies, configuration, and an ephemeral runtime layer. Pin versions and remove unused packages so the image is reproducible, smaller, quicker to move, and easier to audit. Keep large model weights separate when they dominate image size, and cache compiled engines only with the exact hardware and software identity they require.

January 2026 edition snapshot: NVIDIA Inference Microservices (NIMs) package supported model and GPU combinations as ready-made containers, with flexible family images and more specialized model images. They can be a deployable service, reference, or base layer. Choose that opinionated starting point when its tested configuration fits; use a lower-level base when product constraints require control that adaptation would make harder.

Measure the replica envelope before autoscaling. Continuous batching admits new sequences as slots open, but a larger active batch improves aggregate throughput while slowing each request. Set the engine batch limit and autoscaler concurrency target from the same test. Combine traffic with GPU utilization because request count is predictive while utilization reveals expensive sequence shapes. Add a measurement window, minimum and maximum replicas, cooldown, and headroom for forecast error.

A scale-up decision has value only after a replica is ready. Decompose cold start into hardware allocation, image transfer, weight transfer, and engine startup; shrink bytes, place caches near the GPU, and reuse compiled artifacts safely. While capacity arrives, hold work in a bounded first-in-first-out or priority queue. Once ready, route requests by load plus properties such as sequence length, key-value prefix state, or low-rank adaptation (LoRA) weights, and immediately drain queued work up to the new replica's limit.

Above one cluster, separate a global control plane from workload planes that can keep serving independently. Blend reserved capacity for baseline demand with on-demand or interruptible supply for peaks, place workloads near users, and avoid cross-cluster hops inside a multi-stage request. Active-active regions carry live traffic and fail over smoothly; active-passive capacity lowers steady use but makes readiness a critical test. Security policy must cover user data, model weights, infrastructure access, encryption, isolation, retention, and geographic processing constraints.

Deployment is a capacity event. Begin with manual and load tests, then shadow a sample of real traffic. A canary starts the new version at small volume, observes correctness, latency, errors, quality, and scaling, then increases traffic in guarded steps. Keep rollback available and prewarm enough replicas for each step; otherwise the canary measures a cold queue rather than the new service. This avoids the full duplicate GPU fleet required by a blue-green cutover.

Evaluate cost over a representative week. Compare an application programming interface (API) bill for input, output, and cache behavior with dedicated hardware cost under actual utilization, batch, and sequence distributions. Include engineering and operating labor in TCO. Observe request volume, sequence sizes, response codes, percentile latency, replica states, host and accelerator utilization, and queue depth together; logs and change audits explain the correlated movement and support rollback.

Treat the client as part of inference. Reuse transport layer security (TLS) sessions rather than paying setup on every request; for a 300-millisecond P95 objective, a TLS handshake can consume at least ten percent of the budget. Use synchronous Hypertext Transfer Protocol (HTTP) for bounded calls, asynchronous jobs and webhooks for long throughput work, WebSockets for continuous less-structured bidirectional streams, and gRPC for schema-defined service communication. WebSockets favor unstructured real-time data, while gRPC adds a predefined schema and validation for structured bidirectional streams.

Consider a burst that fills every replica just after scale-down. The queue preserves accepted work, but its oldest age begins consuming the SLA. Traffic and concurrency trigger scale-up before utilization alone would; the controller estimates whether cold-start completion will arrive in time. Admission closes or lowers priority when maximum capacity cannot meet deadlines. Newly ready replicas are discovered immediately and receive queued work only to their measured limit. After demand falls, cooldown keeps enough capacity through the likely second wave. This scenario tests the entire feedback loop, while a steady-state throughput test exercises only the engine.

Now remove a GPU node and a region while traffic continues. Health automation must stop new placement, cordon the node, replace affected replicas, and move regional demand without violating data-location policy. Existing workload planes keep serving even if the global controller is impaired. Active capacity absorbs the shift; standby capacity proves it is actually ready with correct artifacts, credentials, network policy, and routes. Operators trace interrupted requests and model-state exposure through audit logs. Recovery is complete only when latency, error rate, queue age, capacity margin, and security controls return to bounds, not merely when a replacement process starts; preserve the timeline to improve the runbook.

An incident timeline should connect product, system, and cost signals. A deployment may increase long inputs, which raises prefill utilization, deepens queues, triggers more replicas, lengthens cold starts by competing for bandwidth, and increases both latency and spend. Looking at any chart alone produces the wrong fix. Correlate release identity, request shape, routing choice, cache behavior, queue age, replica lifecycle, client timing, response codes, and cost allocation under one request and time window. Establish the causal chain before choosing rollback, input limits, routing changes, bandwidth, or more capacity. The same record later explains whether dedicated capacity still beats an API once idle headroom, resilience, and engineering labor are included.

Make data handling part of the serving contract. If inputs and outputs are not required after completion, avoid retaining them. If quality review, audit, or training requires storage, define purpose, region, duration, encryption, access, deletion, and incident evidence before collecting anything. Apply the same discipline to model weights, logs, caches, queues, and temporary artifacts because sensitive content can persist outside the primary database. Portability across providers is useful only when every destination enforces the same controls and geographic restrictions.