Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

arXiv cs.LG / 4/6/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies masked diffusion language models (MDLMs), focusing on speeding up sampling that currently requires many full-sequence denoising passes through a large Transformer.
It proposes “model scheduling,” using a smaller MDLM to replace the full model at selected denoising steps to reduce compute while preserving quality.
Experiments on OpenWebText show early and late denoising steps are more robust to small-model replacement than middle steps, enabling up to a 17% FLOPs reduction with only modest loss in generative perplexity.
The authors back these results with step-importance analyses (loss and KL divergence across timesteps) and an exhaustive search over coarse step segments, concluding the middle of the diffusion trajectory is most sensitive.
Overall, the work suggests architecture-agnostic scheduling rules can accelerate MDLM inference without substantially harming generation quality as measured by perplexity.

Abstract

Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full-sequence denoising passes with a large Transformer and, unlike autoregressive decoding, cannot benefit from KV caching. In this work, we exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps. On OpenWebText, we show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs with only modest degradation in generative perplexity. We support these findings with a step-importance analysis based on loss and KL divergence between small and large models across timesteps, as well as an exhaustive search over coarse step segments, both of which identify the middle of the diffusion trajectory as most sensitive. Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality as measured by generative perplexity.