Shortcuts that enable depth
A residual connection (or skip connection) adds the input directly to the output of a layer, bypassing the learned transformation. Instead of y = f(x), residual blocks compute y = f(x) + x. This simple addition has profound effects. Gradients can now flow directly through the shortcut path, ensuring a gradient magnitude of at least 1. Deep networks (ResNet trained on 152 layers) became practical after residual connections were introduced. The identity path also serves as a highway for feature information: the model learns to add refined representations to the original features rather than replace them.
From ResNet to modern architectures
ResNet proved that residual connections unlock depth, and the pattern became a cornerstone of modern architecture design. Transformers use residual connections around every attention and feedforward block. Dense connections (DenseNet) extend the idea by concatenating outputs from all previous layers instead of just adding the input. Attention mechanisms also implicitly provide skip connections through the residual path before the feedforward. The prevalence of residual connections across very different architectures (CNNs, RNNs, transformers) suggests they capture a fundamental principle about learning in deep networks: representations benefit from additive refinement, and gradient flow requires low-cost paths through depth. Most modern models beyond 20 layers rely on skip connections.