KV Cache Compression: Evaluating SnapKV, H2O, and StreamingLLM for Low-Memory Generation
The KV Cache Memory Crisis in Long-Context Serving
To generate text autoregressively without re-computing past tokens at every step, transformers store the Key and Value activations for every past token in GPU High Bandwidth Memory—a structure known as the KV Cache.
As context windows expand to 32k, 128k, and 1 million tokens, the KV Cache rapidly outgrows the model parameters themselves: $$ ext{Memory}_{ ext{KV}} = 2 imes B imes L imes H imes D imes P$$ Where $B$ is batch size, $L$ is sequence length, $H$ is head count, $D$ is head dimension, and $P$ is byte precision.
For a 70B parameter model serving a batch of 8 users at 128k tokens context in FP16: - Model Parameters: ~140 GB VRAM. - KV Cache Size: Over 320 GB VRAM! - To serve just 8 concurrent users, the KV Cache requires more than double the memory of the model itself.
In 2024, researchers addressed this bottleneck through KV Cache Compression: identifying and retaining only the critical attention tokens while evicting redundant activations from GPU memory.
Three Paradigms of KV Cache Compression
``` 1. StreamingLLM (Xiao et al.): [ Initial Attention Sinks (First 4 tokens) ] + [ Sliding Window Local Cache (Recent 1024 tokens) ] (Evicts all middle tokens -> Enables infinite sequence streaming)
2. Heavy Hitter Oracle - H2O (Zhang et al.): Accumulates Attention Scores over time -> Retains Top-K most-attended "Heavy Hitter" tokens + Local Window.
3. SnapKV (Li et al.): Analyzes Observation Window at the end of prompt -> Selects critical feature heads -> Compresses 80%+ of prompt KV. ```
#### 1. StreamingLLM: The Discovery of Attention Sinks In late 2023, MIT researchers (Xiao et al.) uncovered a surprising quirk of transformer attention: the very first 1 to 4 tokens in any sequence receive an enormous, disproportionate amount of attention score, regardless of their linguistic meaning. - If you evict token #0, the model's perplexity immediately collapses into gibberish. - StreamingLLM preserves just the initial 4 Attention Sink tokens alongside a sliding window of the most recent $K$ tokens (e.g., last 1,024 tokens). - Result: Enables language models to generate infinite text streams (4 million+ tokens) without out-of-memory errors, at constant memory overhead.
#### 2. H2O: Heavy Hitter Oracle While StreamingLLM works well for continuous rolling chat, it fails on document QA because critical factual answers located in the middle of a document are permanently evicted by the sliding window.
H2O (Zhang et al.) dynamically tracks cumulative attention scores: - Tokens that are repeatedly referenced by subsequent queries are classified as Heavy Hitters ($H_2$). - The cache retains these Heavy Hitters plus a recent local window, evicting low-attention filler words (*"the"*, *"and"*, *"however"*). - Memory Reduction: Compresses KV cache by up to 80% while preserving over 95% of factual task performance.
#### 3. SnapKV: Prompt-Aware Observation Windows SnapKV (Li et al., 2024) recognized that during long-context document processing, each attention head consistently attends to specific clusters of tokens. - SnapKV monitors an Observation Window (the final 32 to 64 tokens of the prompt). - It identifies the exact token indices that receive high attention weights across each head. - It compresses the prior document cache down to just those targeted indices, pruning the rest before generation begins.
Benchmark Performance Comparison
| Compression Algorithm | Typical KV Cache Reduction | Perplexity Retention on LongBench | Requires Model Retraining | Primary Use Case | | :--- | :---: | :---: | :---: | :---: | | Full KV Cache (Uncompressed)| 0% (Baseline) | 100% | No | Maximum accuracy, low concurrency | | StreamingLLM | 85% - 95% | Moderate (~78%) | No | Infinite chat agents, live streaming audio | | H2O (Heavy Hitters) | 70% - 80% | High (~92%) | No | Long document QA, summarizing reports | | SnapKV (Observation Window)| 75% - 85% | Very High (~96%) | No | Complex RAG synthesis, 100k+ enterprise lookups |
Synergy with Modern Serving Engines
KV Cache Compression works hand-in-glove with memory management systems: - While vLLM PagedAttention eliminates internal memory fragmentation through virtual memory paging, KV compression algorithms reduce the actual volume of tokens that need to be paged. - Together, they allow a single 8xH100 server to scale concurrent long-context user capacity from 8 streams to over 64 streams, dramatically improving enterprise serving margins.
To learn more about serving infrastructure economics, explore our analysis on The Economics of AI Tokens.