Gradient magnitudes in the chain rule
Backpropagation chains gradients through layers via repeated multiplication. If the gradient at each layer is 0.9, then after 50 layers it is 0.9^50, essentially zero. Early layers receive negligible updates and stop learning. Conversely, if each layer multiplies by 2, the gradient becomes 2^50, a trillion-fold explosion that causes massive weight updates and training divergence. The problem stems from the chain rule: a network with many layers naturally multiplies gradients many times. The magnitude depends on activation functions (sigmoids have gradients bounded by 0.25, dangerous for deep networks) and weight matrices (multiplying by the same weight repeatedly amplifies or attenuates gradients). Vanilla RNNs suffer acutely because they reuse the same weight matrix across timesteps, amplifying this effect over sequence length.
Solutions and modern stabilization
Gradient clipping caps gradient norm to prevent explosions, a simple fix that works. Careful initialization (like He initialization) sets up weights so gradients neither shrink nor explode early in training. Normalization layers (batch norm, layer norm) keep activations in reasonable ranges, preventing extreme gradient values. Residual connections provide skip paths where gradients flow with magnitude 1, solving the depth problem. Modern architectures combine all of these, enabling training of networks 100+ layers deep. The field learned that deep networks are not inherently impossible to train; they just require careful engineering. Understanding vanishing/exploding gradients is key to diagnosing training failures in modern networks.