How gradients flow backward through layers
Backpropagation is the algorithm that trains neural networks. During a forward pass, activations flow layer by layer from input to output, eventually producing a loss. The backward pass inverts this flow: gradients flow from the loss backward toward the input. At each layer, the chain rule breaks the gradient of the loss with respect to that layer's weights into the gradient of the output with respect to the weights times the gradient of the loss with respect to the output. This product is computed at each layer, updating the weights in the direction that reduces loss most steeply.
Why understanding gradient flow matters
Gradients compound through layers, which can cause them to vanish (become very small) or explode (become very large) in deep networks. Vanishing gradients mean early layers barely update, so the model cannot learn meaningful low-level patterns. Exploding gradients can cause weight updates to overshoot and destabilize training. Modern architecture choices like residual connections and layer normalization exist to stabilize gradient flow across depth. Understanding backprop also explains why activation function choice matters: activations with large gradients in the active region allow information to flow; those with tiny gradients create bottlenecks.