Sharding a Model
Models and context windows have outgrown the memory of any single GPU, so a large model must be split across many. Three strategies — pipeline, tensor, and expert parallelism — split the model in different ways and pay for it with different communication patterns.
The unit of splitting differs in each strategy. Pipeline parallelism puts whole
layers on different GPUs. Tensor parallelism splits a single matrix multiply
within a layer across GPUs. Expert parallelism places different MoE
experts on different GPUs. The interconnect available — fast NVLink inside a
node (~1 TB/s) vs. a slower network between nodes — heavily shapes which strategy you reach for.
Pipeline parallelism — layers split across GPUs, and the bubble
step-throughEach GPU holds a contiguous slice of layers. Microbatches flow stage→stage; watch the idle pipeline bubble shrink as more microbatches keep every GPU busy.
A single request must traverse every GPU, and only one stage works on it at a time — so
pipelining adds latency and, on its own, leaves GPUs idle during fill and drain. The communication is
gentle though: each stage only forwards activations to the one next GPU
(point-to-point). That low communication-to-compute ratio is why pipeline parallelism is
favored across nodes, where the interconnect is slower.
Tensor parallelism — one matmul split across GPUs + AllReduce
step-throughA single big matrix multiply (an FFN or attention projection) is sharded by columns. Each GPU computes a partial result on its weight shard, then an AllReduce combines the partials so every GPU ends up with the full layer output. This happens inside every layer.
NVLink
that's worth it: every GPU works simultaneously, lowering request latency. It's also the
only option when even a single layer is too big to fit on one GPU.
Expert parallelism — routing MoE tokens with all-to-all
auto-playIn a Mixture-of-Experts layer the router picks the top-k experts per token.
Experts live on different GPUs, so an all-to-all dispatch ships each token to its expert's GPU;
the expert FFN runs; then a combine returns results home before layer norm proceeds.
Only the top-k experts fire per token, and an expert is either fully active or idle —
which makes one expert a tidy unit to place on one GPU. But because each token activates
different experts on different GPUs, dispatch is cluster-wide all-to-all traffic.
A single overloaded expert (a load imbalance) stalls the whole inference pass — everyone waits
for the slowest expert before combine.
Three strategies, side by side — and combined
step-throughWhat each strategy splits, its communication pattern and cost — and how real deployments stack them: tensor parallelism within a node, pipeline parallelism across nodes.