Adding order to orderless attention
Attention mechanisms are inherently permutation-invariant: they compute relationships between all pairs of tokens without regard to sequence position. While this flexibility is powerful, it also means the model cannot distinguish between identical sequences presented in different orders. Positional encoding solves this by injecting position information directly into token embeddings before they enter the attention layers.
The classical approach uses sinusoidal functions of different frequencies to encode each position and embedding dimension, creating a unique fixed pattern for every position. Learnable positional embeddings offer an alternative, allowing the model to optimize position representations during training. Both methods create a structured signal that attention can parse to extract sequential relationships.
Rotary embeddings for extrapolation
Rotary Position Embedding (RoPE) takes a different approach by applying geometric rotations to the query and key vectors before computing attention scores. This method naturally handles sequences longer than those seen during training, since the rotation angles scale predictably with position. RoPE has become standard in modern large language models because it combines stability with strong extrapolation properties, allowing models trained on sequences of length 2048 to handle 8192-token contexts at inference time.