Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding

arXiv cs.CL / 5/4/2026

📰 NewsDeveloper Stack & InfrastructureModels & Research

共有:

Key Points

Tree-based speculative decoding can lose efficiency on sparse MoE models because larger draft trees activate more experts, increasing verification cost on the target side.
The paper introduces EVICT, a training-free, hyperparameter-free, lossless method that truncates the draft tree before target verification to keep only the most cost-effective prefix.
EVICT uses fine-grained “drafter” signals to estimate whether candidate tokens are likely beneficial, and combines these estimates with verification-cost profiles measured offline.
Experiments across multiple MoE backbones and benchmarks show EVICT can deliver up to 2.35× speedup over autoregressive decoding and about 1.21× over the SOTA baseline EAGLE-3, while reducing unnecessary expert activations.
EVICT is designed to integrate well with high-performance graph-based serving via SGLang, supporting practical deployment in existing inference stacks.

Abstract

Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (MoE) models. As the draft tree grows, different branches activate different experts, expanding the union of activated experts and substantially increasing target-side verification cost. We propose EVICT, a training-free, hyperparameter-free, and lossless adaptive verification method for MoE speculative decoding. EVICT makes every verified token count by truncating the draft tree before target verification and retaining only the cost-effective prefix. It leverages fine-grained drafter signals to estimate candidate benefit, combines them with offline-profiled verification cost, and remains highly compatible with the high-performance graph-based serving framework SGLang. Extensive experiments on diverse MoE backbones and benchmarks show that EVICT achieves up to 2.35x speedup over autoregressive decoding and an average 1.21x speedup over the state-of-the-art baseline EAGLE-3, while significantly reducing unnecessary expert activations during verification.