Hardware

Inference hardware is a chain of finite resources, not a single speed number. Tensor Core throughput limits compute-heavy work such as long-prompt prefill and media generation. Video random-access memory (VRAM) capacity decides whether weights and working state fit, while VRAM bandwidth often limits autoregressive decode. Once a model spans devices, interconnect bandwidth and synchronization join the critical path. Select hardware by the phase and resource that actually saturate under production-shaped traffic.

Picture every inference step as a conveyor: values are fetched from a storage tier, transformed by compute units, and sometimes exchanged with another device. The fastest compute unit still idles if the conveyor cannot supply operands. Small caches are closest and fastest; capacity increases as you move toward VRAM, host memory, and storage, while access cost rises. A balanced system keeps the hottest working set near compute and minimizes crossings over slower boundaries.

Consider a language service with long prompts and short answers. Estimate weights from parameter count and precision, then add the key-value cache, activations, and runtime buffers required by the longest sequence, target concurrency, and batch. Confirm peak headroom before comparing headline speed. Benchmark prefill and decode separately: processing the full prompt may saturate Tensor Cores, while token-by-token generation may saturate VRAM bandwidth. A device with more arithmetic but less capacity or bandwidth may win the first-token path yet lose steady decode or fail with an out-of-memory (OOM) error at long context. Compare the whole workload envelope, including tail latency across sequence and concurrency distributions, rather than naming a winner from a specification table.

Now let the model exceed one device. Within a node, fast GPU links can exchange partial results often enough for a layer-synchronous split to remain useful. Across a second node, each synchronization crosses a slower fabric, and accumulated communication can erase the added compute. The same eight GPUs can enlarge one replica or form an independent replica. Enlarge only when model capacity, key-value cache headroom, or per-request latency requires it; otherwise replication avoids cross-node synchronization and usually serves more independent requests. Model size, cache headroom, concurrency, and the algorithm's communication pattern must jointly determine the topology, not GPU count alone.

Placement extends the hierarchy beyond the rack. A phone can handle a small, immediate, privacy-sensitive request without a network round trip; a workstation can use large unified memory or a high-end discrete GPU; a datacenter can combine powerful accelerators, sustained cooling, and standardized networking behind a scalable service. Alternative chips may target memory bandwidth, energy efficiency, or a specialized dataflow. The engineering question is not which location or vendor wins absolutely, but which stage can run reliably within local compute, capacity, sustained thermals, battery, software support, and available supply. A hybrid system also needs capability detection, a clear data boundary, and a fallback to central service when an edge device is unsupported or inadequate.

A procurement comparison must normalize complete instances, not chip names. Find the smallest instance shape that fits weights plus serving headroom; include host CPU, host memory, storage, GPU form factor, network, and interconnect; and measure sustained performance with the same driver, runtime, engine, precision, and request distribution. Convert price into useful tokens per second or cost per thousand requests while meeting the latency objective instead of dividing a theoretical peak by price. Record regional capacity, startup time, reservations, replacement behavior, and multi-instance partitioning. A suitable modern slice can beat an underused full GPU for a small model, while a whole device preserves burst and cache headroom. Scarce supply can invalidate an excellent benchmark if the required topology cannot be obtained where and when the service must scale.

Design for failure at every boundary. When VRAM is tight, limit admission, lower batch, shorten context, or choose a smaller model before OOM crashes a replica. When an intra-node or cross-node link degrades, identify synchronization delay and switch to a lower-communication configuration or independent replicas. When an edge device lacks support, overheats, runs low on battery, or responds too slowly, use an explicit cloud fallback while preserving consent and data boundaries. Monitoring must distinguish Tensor Core saturation, VRAM capacity pressure, memory-bandwidth stalls, interconnect congestion, thermal throttling, model-loading input and output, and ordinary queueing; otherwise every failure appears only as higher latency. A hardware choice is complete only when these degradation modes are observable, bounded, and recoverable.

Keep an auditable hardware decision record: production input, output, and concurrency distributions; weight and cache arithmetic; device and node topology; complete instance specifications; driver and engine versions; phase benchmarks; price basis; capacity source; and rejected options. State assumptions such as precision, longest context, acceptable P99, expected cache hits, and cross-node traffic. Rerun the same tests when traffic, model size, precision, kernels, cloud supply, or pricing changes. If one assumption changes, first retest its affected phase, then confirm end to end that the bottleneck did not move elsewhere. Treat a provider's host or network change as a new candidate even when the GPU name is unchanged. Preserve test date and edition so historical results are not mistaken for current guarantees; the selection is an engineering conclusion under explicit assumptions, not a timeless product ranking.