Speculative Decoding at Scale: Comparing Medusa, EAGLE, and Draft Model Inference Latency

The Memory Wall of Autoregressive Token Generation

In Large Language Models, inference is divided into two distinct operational phases: 1. Prefill Phase: The model processes the input prompt all at once. Because all input tokens are known in advance, matrix multiplications are densely parallelized across GPU Tensor Cores, achieving high hardware utilization. 2. Decode Phase: The model generates one single token at a time autoregressively. Each new token requires reading all billions of model weights from GPU High Bandwidth Memory (HBM) into on-chip cache registers.

Because decoding a single token performs minimal arithmetic per byte transferred, the GPU sits heavily memory-bandwidth bound. When generating text for an interactive user, modern GPUs operate at less than 15% of their theoretical compute capacity.

In 2023 and 2024, the AI systems research community solved this bottleneck using Speculative Decoding: generating multiple candidate tokens in parallel and verifying them in a single forward pass, unlocking 2x to 3x sustained speedups with zero loss in mathematical accuracy.

The Core Principle: Speculate Fast, Verify in Parallel

Speculative Decoding exploits a fundamental computational asymmetry: generating $K$ sequential tokens autoregressively takes $K$ separate memory read cycles, but verifying $K$ candidate tokens simultaneously takes only ONE forward pass.

``` Step 1: Speculation (Fast Draft Mechanism) Draft Engine predicts next 4 tokens: [ "the", "capital", "of", "France" ] Step 2: Parallel Verification (Target Model) Target model evaluates all 4 candidates in a single GPU forward pass: - "the" -> Target Probability: 98% [ACCEPTED] - "capital" -> Target Probability: 95% [ACCEPTED] - "of" -> Target Probability: 99% [ACCEPTED] - "France" -> Target Probability: 97% [ACCEPTED] + Target Model emits bonus 5th token: [ "is" ]

Result: 5 tokens generated in the time of 1 standard forward pass! ```

If the target model rejects candidate token #3, it discards tokens #3 and #4, accepts tokens #1 and #2, samples the correct token #3 from its own distribution, and restarts. Crucially, through rejection sampling, the final token distribution is mathematically identical to running the target model standalone.

Three Generations of Speculative Architectures Compared

``` 1. Standard Draft Model (Leviathan et al.): [ Small Draft LLM (e.g. Llama-3-8B) ] ──> Draft Tokens ──> [ Large Target LLM (Llama-3-70B) ]

2. Medusa (Cai et al.): [ Target LLM Hidden States ] ──> [ Head 1 (+1) ] [ Head 2 (+2) ] [ Head 3 (+3) ] (Tree Attention)

3. EAGLE (Li et al.): [ Target LLM Top-Layer Features ] + [ Token Embeddings ] ──> [ Lightweight Transformer Decoder ] ```

#### 1. Standard Draft-Model Speculation A small, fast language model (e.g., Llama-3-8B) acts as the draft generator for a flagship model (Llama-3-70B). - Limitation: Both models must reside in GPU VRAM simultaneously. If the draft model's vocabulary or tokenizer differs, distillation alignment degrades acceptance rates.

#### 2. Medusa: Eliminating the Draft Model Medusa replaces the auxiliary draft model with multiple lightweight feed-forward heads attached directly to the final layer of the base model: - Head 1 predicts token $t+1$. - Head 2 predicts token $t+2$. - Head 3 predicts token $t+3$. - Medusa constructs a Candidate Tree of possible multi-token sequences and uses a custom Tree Attention Mask to evaluate all combinations concurrently in a single forward pass. - Advantage: Requires zero additional model weights in VRAM (Medusa heads add less than 1% parameter overhead).

#### 3. EAGLE: Feature-Level Speculation EAGLE (Extrapolation Algorithm for Greater Language-model Execution) addresses Medusa's limitation: predicting multiple tokens ahead without understanding semantic context causes high error rates on complex reasoning. - EAGLE operates in the feature vector space rather than the discrete token space. - It passes the target model's second-to-top hidden state through a single lightweight transformer layer, capturing complex inter-token dependencies. - Result: Boosts average token acceptance length from ~2.2 tokens to 3.5+ tokens per forward pass.

Benchmark Speedups Across Production Workloads

| Speculative Method | Extra VRAM Overhead | Average Acceptance Length | Throughput Speedup | Serving Engine Support | | :--- | :---: | :---: | :---: | :---: | | Baseline (Autoregressive) | None | 1.0 token / step | 1.0x (Baseline) | Universal | | Draft Model (8B $ o$ 70B) | High (~16 GB VRAM) | 2.1 tokens / step | 1.8x - 2.1x | vLLM, TGI, SGLang | | Medusa (Tree Attention)| Negligible (< 1 GB)| 2.3 tokens / step | 2.2x - 2.6x | vLLM, TensorRT-LLM | | EAGLE-2 (Feature SDE) | Low (< 2 GB) | 3.4 tokens / step | 2.8x - 3.2x | SGLang, vLLM |

On coding and structured JSON generation (where code syntax is highly predictable), EAGLE routinely reaches over 4 accepted tokens per step, tripling generation speed without modifying hardware.

Production Deployment Guidelines

1. Serve on High-Throughput Engines: Frameworks like vLLM and SGLang support speculative decoding natively with zero code changes. 2. Combine with KV Cache Paging: Pair speculative verification with vLLM PagedAttention to prevent tree-attention KV memory spikes. 3. Target Predictable Domains: Speculative decoding delivers its highest speedups in coding, unit testing, and structured outputs (see Guaranteed JSON Schema: Constrained Decoding).

Open-source reference implementations are accessible on GitHub at FasterDecoding/Medusa ↗ and SafeAILab/EAGLE ↗.