Parallel representation learning in transformers
Multi-head attention enables a transformer to simultaneously learn different types of relationships within the same sequence. Instead of using a single attention mechanism, the model maintains multiple independent sets of query, key, and value projections (heads), each operating in a lower-dimensional space. These heads run in parallel and capture distinct semantic patterns: one head might focus on syntactic dependencies while another tracks long-range semantic links.
The outputs from all heads are concatenated and linearly combined, effectively allowing the model to blend multiple learned representations into a single downstream layer. This redundancy provides both regularization and expressiveness, letting the transformer explore more of its parameter space during training.
Why multiple perspectives outperform single projection
A single attention head is constrained by its embedding dimension and cannot efficiently model all types of relationships simultaneously. By distributing the task across 8, 12, or more heads, the transformer can specialize: some heads become position encoders, others capture grammatical structure, and still others learn task-specific patterns. This division of labor improves gradient flow during backpropagation and makes the model more robust to different input distributions.
Empirically, removing heads from a trained model consistently degrades performance, confirming that each head learns complementary information rather than redundant patterns.