Why gradients degrade in very deep networks
In a deep network, gradients propagate backward by multiplying local gradients at each layer. If each layer's gradient (the derivative of output with respect to input) is less than 1, these products shrink exponentially as you go deeper. After 50 layers, a gradient of 0.98 per layer becomes 0.98^50, effectively zero. Early layers see negligible gradients and barely update, unable to learn meaningful patterns. Conversely, if gradients are greater than 1, they can explode, causing unstable training. This tradeoff between vanishing and exploding gradients is a fundamental challenge in depth.
Architecture solutions that enable scaling
Modern networks scale depth by managing gradient flow carefully. Residual connections (used in ResNet) add a shortcut that lets gradients bypass the nonlinearity, ensuring a gradient of at least 1 through the skip path. Layer normalization (used in transformers) keeps activations in a region where gradients are large. Careful initialization (e.g., He initialization for ReLU) sets up the initial gradient magnitudes to avoid extreme values. These architectural choices are not optional tweaks for very deep networks; they are essential. Without them, networks deeper than about 20 layers become impossible to train. Modern language models (100+ layers) would not be possible without these gradient-flow innovations.