Normalization across features, not batches
Layer normalization normalizes each example independently over the feature dimension. For a batch of 32 examples with 768 features, layer norm computes the mean and variance of the 768 values within each example and normalizes. This differs from batch norm, which pools statistics across the 32 examples. Layer norm is invariant to batch size: the same example produces the same normalized representation whether you run it alone or with a thousand others. This property makes it ideal for scenarios with variable batch sizes, small batches, or at inference time where batch statistics are meaningless.
Why transformers standardized on layer norm
Transformers use layer norm because it stabilizes training, permits higher learning rates, and is deterministic at inference time (no train-test mismatch like batch norm). The normalization happens after the linear projection, creating a stable distribution for downstream attention and feedforward operations. Empirically, layer norm enables training very deep transformer models (100+ layers) without gradient collapse. Batch norm was tried in early transformer experiments and underperformed. Layer norm became the de facto standard for transformer architectures and has since been adopted in other modern networks. Some variants (like root mean square layer norm in recent models) simplify the computation further while maintaining the same benefits.