Beyond Vanilla Speculative Decoding: Medusa, Eagle, and Lookahead

The Problem Vanilla Speculative Decoding Didn’t Solve

Leviathan et al. (2023) introduced a beautiful idea: use a cheap draft model to propose multiple tokens at once, then verify them in a single forward pass of the large target model. The mathematics is elegant — if the draft model’s token distribution closely approximates the target’s, you get rejection sampling that is statistically equivalent to the original model’s output, at a fraction of the sequential decoding cost.

The speedup is real. On a 7B target with a well-matched 68M draft, Leviathan demonstrated 2–3× wall-clock improvement without any change to output quality. The mechanism is sound.

The deployment story is not.

Vanilla speculative decoding’s central operational burden is the draft model itself. It must be a separate model — independently trained, independently served, independently versioned. And it cannot be just any small model: it must be closely aligned with the target’s distribution. In practice, that means it needs to be a member of the same model family, trained on the same data, with the same tokenizer, and ideally fine-tuned on the same instruction distribution if the target has been fine-tuned.

The consequences compound:

• Every time you fine-tune your target model — for a new domain, a new customer, a new RLHF iteration — your draft model is now misaligned. Its token-acceptance rate drops from 80%+ to something that may no longer justify the batching overhead, and you must either re-fine-tune the draft or revert to standard autoregression.
• Memory pressure doubles. The draft model lives in HBM alongside the target. On an H100 with 80 GB, a Llama-3-70B target in FP8 (~35 GB) plus a 7B draft (~3.5 GB) is manageable. On a 70B target in BF16 (~140 GB) across a tensor-parallel 8-GPU node, any additional model increases NVLink traffic and complicates CTA scheduling.
• Serving infrastructure must manage two inference engines — two sets of CUDA contexts, two sets of KV caches, two warm-up paths.

This is the thorn that motivated the three approaches that now dominate production speculative decoding practice: Medusa, Eagle, and Lookahead Decoding. Each attacks the draft-model deployment problem from a different angle, and each makes a different trade.

Medusa: Heads, Not Models

Cai et al. (arXiv:2401.10774) reframe the question entirely. Rather than maintaining a separate draft model, why not attach additional decoding heads directly to the target model, each head trained to predict tokens at offset positions 1, 2, …, k ahead?

Architecturally, Medusa extends the target’s existing LM head — a linear projection from the final hidden state to vocabulary logits — with k parallel sibling heads. Head 1 predicts the token at position t+1, head 2 predicts t+2, and so on, all reading from the same final hidden state produced by the base model’s transformer stack. No additional transformer layers are introduced. The base model’s weights are frozen; only the new heads are trained.

The training objective is straightforward: cross-entropy over the offset-k ground-truth tokens. In practice Medusa heads are fine-tuned on a few thousand GPU-hours of target-model outputs, not raw pretraining data — a cost that is roughly proportional to the cost of a LoRA fine-tune, not a pretraining run.

Verification uses tree-based speculative decoding. Rather than greedily committing to a single k-token draft, Medusa samples multiple candidates from each head’s distribution and constructs a tree of possible continuations — branching by temperature-sampled candidates at each position. The target model verifies the entire tree in a single forward pass using a custom tree attention mask that allows simultaneous evaluation of all candidate paths without causal leakage. The longest accepted prefix of any valid branch is committed.

The results are competitive: Medusa reports roughly 2–3× speedup on standard benchmarks against sequential decoding of the same target model. Acceptance rate varies with head depth — the first head accepts at high rates (similar to a well-matched external draft), while heads 4 and 5 accept considerably less often, which is why tree-based sampling rather than greedy decoding from each head is critical to extracting the full benefit.

The deployment story is the key advantage: because the draft heads are attached to the target model, fine-tuning the target is equivalent to re-fine-tuning Medusa — you simply include the Medusa heads in the fine-tuning pass with their own learning rate. Distribution drift between draft and target is structurally impossible.

The limitation is equally structural: Medusa predicts future tokens from the current hidden state, which does not encode the predicted intermediate tokens. Head 2’s prediction of t+2 cannot condition on the actual token at t+1, only on the last observed state. This conditional independence assumption is the ceiling on Medusa’s acceptance rate and explains the gap relative to Eagle.

Eagle: Drafting in Feature Space

Li et al. (arXiv:2401.15077, Eagle-2 in arXiv:2406.16858) identified the root cause of Medusa’s acceptance-rate ceiling and engineered a direct solution.

The insight is precise: draft quality degrades when the draft distribution is misaligned with the target’s conditional distribution given the true token sequence. Medusa’s heads predict p(t+k | h_t), where h_t is the hidden state at the last observed token. The actual target distribution is p(t+k | t+1, t+2, …, t+k−1, h_t) — conditioned on the intermediate tokens that were generated in the draft. Since Medusa cannot access those tokens during drafting, its conditionals are necessarily approximate.

Eagle’s solution is to draft autoregressively in the feature space of the target model rather than in token space. Eagle trains a lightweight transformer layer — typically a single-layer auto-regressive network — that takes as input the concatenation of the target’s last hidden state at each step and the embedded draft token predicted in the previous step. It then predicts the next hidden state, from which an LM head (shared with the target model) reads the next draft token. The key: each draft step conditions on the previous draft’s feature vector, not just the surface token, so the conditional structure is correctly maintained through the draft chain.

This seemingly small architectural change has a large empirical consequence. Eagle reports acceptance rates above 80% — substantially higher than Medusa — because the draft model’s conditional distribution more faithfully mirrors the target’s. The speedup is correspondingly larger: Eagle demonstrated roughly 3–4× wall-clock improvement on Vicuna-13B and Llama-2-70B in their benchmarks, outperforming Medusa by a meaningful margin on the same hardware.

Eagle-2 (Li et al., arXiv:2406.16858) adds dynamic draft trees. The original Eagle uses a fixed tree structure for verification — a predetermined branching factor and depth. Eagle-2 instead builds the draft tree adaptively at runtime, expanding branches according to a confidence score derived from each node’s predicted feature distribution. Low-confidence branches are pruned; high-confidence branches are expanded. This reduces wasted verification flops on branches the model would never accept, and improves average token generation per step by roughly 20% relative to Eagle-1 on the same model family.

The cost is training complexity. Eagle’s lightweight drafting network must be trained with access to the target model’s internal representations — specifically, the sequence of hidden states produced by the target over a training corpus. This requires running the full target model over the training data to collect activations, then training the Eagle network on those activations. It is more expensive than training Medusa heads, and it more tightly couples the drafting network to a specific model checkpoint. A change in target model architecture or a major weight update requires re-collecting activations and retraining.

Lookahead Decoding: No Draft Model at All

Fu et al. (arXiv:2402.02057) take the most radical position: eliminate the draft model entirely, including any attached heads.

Lookahead Decoding is inspired by Jacobi iteration — a classical technique for solving systems of equations by iteratively updating all unknowns in parallel from the previous iteration’s values. Applied to autoregressive generation: rather than generating tokens strictly left-to-right, maintain a rolling window of n future token positions and speculatively fill them in parallel at each step, using a cache of past n-gram observations (called the n-gram pool) to propose candidates.

Concretely, Lookahead operates in two branches per forward pass:

1. The lookahead branch speculatively fills positions t+1 through t+W in parallel, each position seeded from n-grams observed in previous steps. These predictions are not expected to be correct; they are Jacobi-style guesses that will converge given enough iterations.

2. The verification branch simultaneously checks whether any complete n-gram from the n-gram pool exactly matches a sequence the model would have generated autoregressively starting from position t. Matching n-grams are committed as accepted tokens.

The n-gram pool is populated incrementally as generation proceeds, using outputs from both branches. No external data, no fine-tuning, no training of any kind is required. The method is applied at inference time to any model, with no modification to model weights or architecture.

Speedup is model- and domain-dependent: Fu et al. report roughly 1.5–2.5× improvement on standard benchmarks, reliably lower than Medusa or Eagle on the same model. The acceptance rate of n-gram candidates is highly sensitive to the repetitiveness of the generated text — code generation, where n-gram patterns are common and predictable, sees higher speedup than open-ended chat, where the n-gram pool is sparse and matches are rare.

The value proposition is not raw speed. It is zero deployment friction. Lookahead can be applied to any model behind a standard inference stack without model changes, without training, and without any assumption about model architecture. For organizations that cannot afford to fine-tune speculative components every time the target model changes — or that serve dozens of fine-tuned variants from a single serving infrastructure — Lookahead’s deployment profile is genuinely distinct from the alternatives.

The Trade Matrix

The acceptance rate and speedup figures above are benchmarked across the Llama and Vicuna families on standard conversational and coding datasets. Results vary with model size, temperature, and domain; always validate on your traffic distribution before committing to production.

Production Decision Logic

The choice among these methods is determined primarily by two orthogonal axes: how frequently your target model changes, and whether you have a training budget for speculative components.

Medusa is the correct choice when you have an already-fine-tuned model and want to add speculative decoding with minimal operational surface area. The heads attach to whatever checkpoint you have, fine-tuning the heads is trivially includable in any future fine-tune job by treating them as additional parameters, and the serving path is a single model file. Organizations running a small number of well-characterized target checkpoints — say, a base model and two or three fine-tuned variants — find that Medusa’s operational profile is nearly equivalent to standard serving. The 2–3× speedup closes most of the gap with Eagle at a fraction of the integration cost, and the tree-based verification is well-supported in both vLLM and TensorRT-LLM.

Eagle and Eagle-2 are the correct choice when you are beginning a new training run and can include the Eagle component from the start, or when you are willing to invest activation collection for a high-value model. The acceptance rate advantage — roughly 10–15 percentage points above Medusa in matched comparisons — compounds over long generations, and Eagle-2’s dynamic tree pruning further improves efficiency at longer output lengths. For production systems where throughput is the primary economic variable (token throughput determines cost-per-million-tokens), the additional speedup of Eagle over Medusa is worth the integration cost when the model is stable. The constraint is that each significant architecture or weight update requires rerunning activation collection and retraining the Eagle network — a cost that amortizes well over a stable model but poorly over a rapidly iterating one.

Lookahead Decoding is the correct choice for zero-touch deployment scenarios. Multi-tenant serving infrastructures that host dozens of fine-tuned variants, inference endpoints where the model checkpoint changes weekly, or serving stacks where any model-side modification is gated behind a lengthy validation process — these environments cannot operationally afford Medusa or Eagle’s training requirements per variant. Lookahead’s speedup ceiling is lower, but its floor on deployment friction is also lower than any alternative. It also has a specific niche in code generation: when serving a coding assistant where output repetition and structural patterns are high (function signatures, boilerplate, import blocks), the n-gram pool fills quickly and acceptance rates approach those of Medusa on general text.

One additional consideration governs all three choices at scale: the interaction with continuous batching. Modern serving runtimes (vLLM’s iteration-level scheduling, TensorRT-LLM’s in-flight batching) schedule requests at token granularity, not request granularity. Speculative decoding with multi-token proposals changes the batch step structure — a verified 4-token draft is not four sequential autoregressive steps but one verification step — and this interacts non-trivially with PagedAttention’s KV cache block allocation and with the scheduler’s estimate of time-to-completion for SLA accounting. Medusa’s integration with vLLM is mature as of mid-2025; Eagle’s integration is available but requires careful configuration of tree attention masks within the serving framework; Lookahead is the least disruptive to the scheduler because its Jacobi window maps naturally onto standard batch step semantics.

Synthesis: The Hierarchy of Constraints

The speculative decoding literature after Leviathan converges on a single underlying principle: the acceptance rate is bounded by how much information the draft distribution shares with the target’s true conditional distribution. Vanilla speculative decoding with a well-matched draft achieves high acceptance because the draft has been trained end-to-end to approximate the target. Eagle achieves high acceptance because it conditions on the target’s own feature vectors, which are the richest available representation of the target’s internal state. Medusa accepts less because its heads condition on a single hidden state without access to intermediate draft tokens. Lookahead accepts the least because n-gram statistics from recent generation history are the coarsest possible proxy for the target’s conditional.

The deployment complexity gradient runs in exactly the opposite direction. More information shared between draft and target requires more coupling between them — more training, more architectural dependency, more operational maintenance. The practitioner’s job is to locate the right position on this curve for their specific serving environment.

For the vast majority of production deployments today, the answer is Medusa for stable fine-tuned models, Eagle for new training runs where the activation collection overhead is a one-time cost, and Lookahead for everything that cannot afford either. Vanilla speculative decoding, despite its theoretical elegance, increasingly belongs in the paper rather than the serving stack.

References

• Cai, T. et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774.
• Li, Y. et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv:2401.15077.
• Li, Y. et al. (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv:2406.16858.
• Fu, Y. et al. (2024). Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. arXiv:2402.02057.
• Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023.

@article{fp4-2606017,
  title   = {Beyond Vanilla Speculative Decoding: Medusa, Eagle, and Lookahead},
  author  = {fp4 editorial desk},
  year    = {2026},
  url     = {https://fp4.dev/algorithm/speculative-decoding-variants/},
  journal = {fp4}
}