Local windows and global tokens for 100k+ context
Full attention at 100k tokens is intractable: N x N is 10 billion elements. Sliding window attention attends only to a local neighborhood (e.g., 2048 tokens), keeping full parallelism but limiting context. To recover long-range dependencies, a few 'sink' or 'global' tokens (e.g., the first 4 tokens or a learned [SINK] token) attend to all positions, while regular tokens attend locally.
This hybrid approach preserves long-range information (global tokens see everything) without the quadratic cost.
Preserving quality at long context
Naively training with local attention on 100k sequences doesn't always preserve quality compared to full attention on short sequences. The global tokens are key: they compress information from the full sequence and serve as bottlenecks for reaching distant context. Proper initialization and training of these global tokens is important.
Many recent models (Llama 2 at 34K context, Claude at 100K+) use variants of sliding window plus global tokens.
Extrapolation and position encoding
Long context requires position encodings that extrapolate beyond training length. Rotary position embeddings (RoPE) and ALiBi (Attention with Linear Biases) both extrapolate smoothly. Some models require position interpolation (scaling down position IDs) or fine-tuning to safely extend context. This is an active area of research.