Compressing K and V to a low-rank latent
Multi-Head Latent Attention (MLA) projects key and value matrices to a low-rank latent space before caching. Instead of caching all N x D_head values for each head, you cache N x D_latent (where D_latent << D_head). During attention, you decompress back to full dimensionality for the attention computation.
The compression is learned end-to-end. The model discovers which parts of K and V are truly needed for each head, discarding the rest.
Cache size reduction and quality
MLA can reduce KV cache size by 50-90%, depending on compression ratio. This enables longer sequences or larger batch sizes. Theoretical analysis suggests that most heads don't need the full D_head dimensions; a low-rank subspace is sufficient.
Quality is maintained because the compression is learned with task supervision. Naive truncation loses quality; learned compression preserves it.
Practical deployment impact
For inference at scale, MLA reduces memory, bandwidth, and latency. Serving 100 concurrent long-context requests with KV cache is expensive; MLA makes it tractable. This is particularly valuable for production large language models where cost per inference is critical.