Going Deeper · Module 8

Inference Deep-Dive

Parallelism for huge models7 min readModule 8 of 8

A final deep-dive for the curious. A trillion-parameter model can’t fit or run on one GPU — so it’s split across many. How you split it is a rich engineering trade-off between raw throughput and how fast each user gets their answer. This is the problem all that NVLink bandwidth exists to solve.

#Four ways to split a model

There are four basic parallelism techniques. Each helps with something and costs something else.

Data parallelism (DP)

Put a full copy of the model on each GPU and feed them different requests. Scales throughput linearly — but every GPU needs enough memory for the whole model.

Tensor parallelism (TP)

Split each layer across GPUs so one request is worked on in parallel. Improves per-user speed, but leans hard on high-bandwidth GPU-to-GPU links — a bottleneck at scale.

Pipeline parallelism (PP)

Give each GPU a different group of layers; a request flows through them in sequence. Distributes the model’s weight, but doesn’t do much for responsiveness.

Expert parallelism (EP)

For MoE models: route each request to the specific experts it needs, on different GPUs. Avoids unnecessary parameters, but needs all-to-all communication and complex routing.

The four building blocks. Real deployments combine them.

#Combining techniques

The techniques compose. Combining expert + pipeline parallelism can roughly double per-user interactivity with minimal throughput loss. Adding tensor + expert + pipeline together can triple GPU throughput without sacrificing responsiveness. Finding the right mix is a huge search — for GPT-1.8T MoE, NVIDIA analyzed more than 2,700 configurations of parallelism and chunk length.

#Prefill, decode, and chunking

Inference has two phases: prefill (reading your prompt — sets time-to-first-token) and decode (writing the answer token by token — sets tokens-per-second). Inflight batching and chunking keep the GPU busy across both.

The chunk-size dial

Larger chunks → fewer prefill steps → quicker first token, but slower overall token rate. Smaller chunks → faster token output, but a longer wait for the first one. There’s no universal best — it depends on the deployment, which is exactly why orchestration software like Dynamo matters.

Source: Appendix A of the brief — “Advanced Parallelism Techniques in AI Inference for Trillion-Parameter Models.”

You made it to the end

You now have the full picture: why reasoning AI needs new hardware, the chip and engines that provide it, the NVLink fabric that scales it, and the systems that ship it. Use the glossary anytime to refresh a term.

Go to glossary →