Why caching K and V speeds up generation
During generation, the model produces one token at a time. Each new token attends to all previous tokens (the context). Computing attention from scratch each time is wasteful: you recompute K and V for all previous tokens even though they haven't changed. KV caching stores the K and V matrices from previous steps, reusing them for new tokens.
The cache grows with each generation step. After generating 100 tokens, the cache contains 100 steps worth of K and V. Each new token only computes its own K and V, attending to the full cached context.
Speedup and memory cost
With KV caching, generation time per token drops from O(n^2) to O(n) (where n is context length). For a 4K context, this is a massive speedup: no KV cache requires 16M attention operations per token, while with cache it's 4K lookups. The trade-off is memory: you must store all K and V from all positions.
For a 70B parameter model with 4K context, KV cache consumes roughly 32-64 GB per sequence. This limits batch size: few sequences fit if KV cache is large.
Cache size management strategies
Sliding window attention discards old cache entries, keeping only the most recent (e.g., last 1K tokens). This trades off context for memory. Sparse attention patterns also reduce cache size by selecting which positions to attend to. PagedAttention (used in vLLM) manages cache like virtual memory, reusing pages and paging out unused cache to disk.