Why bigger batches enable larger learning rates
Gradient descent on a single example is noisy; gradients from a batch of examples are smoother. A smooth gradient estimate lets you take larger steps without overshooting. This is the fundamental insight: batch size increases gradient signal-to-noise ratio, enabling higher learning rates and faster training.
If you double batch size without changing the learning rate, you halve the noise, roughly equivalent to a square-root reduction in variance. This margin can be reinvested in learning rate.
The critical batch size threshold
Scaling batch size does not improve indefinitely. At some point (the critical batch size, often 30-100 times your model size in elements), doubling the batch again provides negligible returns on optimization speed. The law of diminishing returns kicks in because the gradient becomes so smooth that even the maximum stable learning rate doesn't saturate the improvement potential.
Beyond this threshold, you're spending hardware cost for minimal speedup. Modern large-scale training often uses a 'sweet spot' batch size, not the largest possible.
Practical scaling during distributed training
In distributed training, batch size scales with the number of machines. Eight GPUs with batch size 32 each = global batch size 256. This requires careful learning rate tuning: scaling the LR naively can hurt convergence, so teams often use square-root scaling rules or warmup schedules to smooth the transition.