Inside vLLM: How PagedAttention Eliminates KV-Cache Memory Waste in High-Concurrency Serving

The Memory Wall in Large Language Model Serving

In production LLM infrastructure, the primary bottleneck limiting serving concurrency is not compute power—it is GPU High-Bandwidth Memory (HBM) capacity and allocation efficiency.

During autoregressive token generation, language models compute Key and Value tensors for every attention head at each layer. To avoid recalculating these tensors for previous tokens on every generation step, serving systems store them in GPU VRAM inside a structure known as the Key-Value (KV) Cache.

Historically, LLM serving engines suffered from catastrophic memory waste: 60% to 80% of total GPU memory allocated for KV caches was completely wasted. In 2023, a team of researchers from UC Berkeley's LMSYS organization published the landmark PagedAttention paper and released vLLM, revolutionizing production inference economics.

Why Legacy KV Caching Wasted 80% of VRAM

Legacy serving systems allocated memory for KV caches based on two flawed assumptions: 1. Contiguous Allocation: Memory chunks had to be stored in contiguous physical GPU memory addresses. 2. Maximum Length Reservation: Because output token lengths cannot be known in advance, systems pre-allocated memory for the maximum possible sequence length (e.g., 4,096 or 8,192 tokens) for every incoming request.

This led to three severe forms of memory waste: - Internal Fragmentation: A user request generating only 200 tokens reserved 4,096 tokens of VRAM. The remaining 3,896 token slots sat idle and unusable by other requests. - External Fragmentation: Memory was split into fragmented chunks of varying sizes, preventing new requests from finding sufficiently large contiguous blocks. - Reservation Waste: Systems held memory buffers for future generation steps that had not yet occurred.

Because GPU VRAM was exhausted by reserved, empty memory buffers, batch sizes had to remain artificially small (e.g. 8 or 16 concurrent requests on an NVIDIA A100 GPU), leaving tensor cores drastically underutilized.

The PagedAttention Solution: Operating System Paging on GPUs

PagedAttention solves KV-cache fragmentation by borrowing one of the most fundamental concepts in computer science: Virtual Memory Paging.

Just as modern operating systems divide physical RAM into fixed-size pages and map them to non-contiguous virtual address spaces using page tables, PagedAttention: 1. Divides the KV cache of each sequence into fixed-size KV blocks (typically containing 16 or 32 tokens). 2. Allocates KV blocks dynamically into non-contiguous physical GPU memory pages as tokens are generated. 3. Maintains a Block Table for each request that maps logical token positions to physical memory blocks.

#### How It Works During Generation: - When a prompt enters the system, vLLM allocates only enough KV blocks to store the initial prompt tokens. - As the model generates tokens one by one, it fills the current KV block. Only when a block reaches capacity does vLLM allocate a single fresh block from its global pool. - Because blocks do not need to be contiguous in physical memory, external fragmentation drops to zero, and internal fragmentation is confined strictly to the final incomplete block of a sequence (averaging less than 4% memory waste).

Continuous Batching and Chunked Prefill

PagedAttention enabled two critical subsequent innovations in the vLLM engine:

• Continuous (Iteration-Level) Batching: In traditional batching, an entire batch of requests had to wait until the longest request finished generating before discharging memory. With vLLM, completed requests release their physical blocks immediately on the next token iteration, allowing new requests to join the running batch without waiting.
• Chunked Prefill: Processing large prompt contexts (the prefill phase) consumes massive compute spikes that stall active generation (the decode phase), causing severe latency jitter. Chunked prefill slices large prompt contexts into manageable chunks, interleaving them smoothly with ongoing token generation steps.

Impact on Serving Throughput and Cloud Economics

The empirical impact of PagedAttention on production inference infrastructure was staggering: - Batch Size Scaling: Eliminating memory waste allowed GPU servers to handle 4x to 8x more concurrent requests within the same VRAM footprint. - Throughput Gains: Serving throughput increased by 2x to 4x compared to HuggingFace TGI and FasterTransformer without altering model weights or sacrificing perplexity. - Prefix Caching: Identical system prompts across hundreds of concurrent user requests share the exact same physical KV blocks, reducing prompt processing time to near zero.

Today, vLLM powers the inference backends of major cloud providers like Together AI and GPU orchestrators on RunPod.

To explore how model serving architectures influence enterprise API costs, review our analysis on Enterprise LLM Pricing & Cost Optimization.

Summary

PagedAttention proved that the most transformative breakthroughs in artificial intelligence do not always require larger datasets or novel neural architectures. By applying foundational systems engineering principles to GPU memory management, vLLM unlocked massive efficiency gains that underpin the modern generative AI economy.

The original research paper is available at arXiv:2309.06180 ↗, and open-source documentation is accessible at docs.vllm.ai ↗.