Three axes of distributed training
Training large neural networks across many GPUs uses three complementary strategies that each split something different. Data parallelism replicates the full model on every GPU and splits the training batch across them; each device computes gradients on its shard, then gradients are averaged (all-reduce) so all replicas stay in sync. Tensor parallelism splits the individual weight matrices of a single layer across GPUs, so one matrix multiply is computed cooperatively, which is necessary when one layer is too large to fit on a single device.
Pipeline parallelism splits the model by layers, assigning consecutive layer groups (stages) to different GPUs and streaming micro-batches through them like an assembly line.
When each is used
Data parallelism is the simplest and scales batch throughput, but every GPU must hold the whole model. Tensor parallelism is used inside a node where interconnect bandwidth (such as NVLink) is high, because it communicates heavily within each layer. Pipeline parallelism reduces per-GPU memory by holding only some layers, at the cost of pipeline bubbles when stages wait for one another. Large-model training typically combines all three, sometimes called 3D parallelism.