Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

arXiv cs.CL / 5/1/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

Long-context processing remains challenging because standard Transformers have quadratic compute costs and limited ability to extrapolate to much longer sequences.
The paper analyzes chunk-based sparse attention models that are intended for extreme length generalization, aiming to identify what architectural ingredients actually drive their performance.
Using a unified framework and extensive ablation studies, it finds three critical design principles: a non-linear Chunk Encoder with a dedicated CLS token for retrieval, a bypassing residual path to incorporate retrieved global information stably, and enforced sparse selection during pre-training to reduce train–test mismatch.
The authors provide theoretical motivation for why intra-chunk processing and “landmark” generation work, and report new state-of-the-art results for training-free length extrapolation from 4K contexts to 32M tokens on RULER and BABILong.
Overall, the work translates prior chunk-sparse intuition into empirically grounded, reusable engineering principles for building next-generation long-context language models.

Abstract

Effectively processing long contexts is a critical challenge for language models. While standard Transformers are limited by quadratic complexity and poor length extrapolation, alternative architectures like sliding window attention and state space models sacrifice the ability to effectively utilize the full context due to their fixed-size memory. Chunk-based sparse attention has emerged as a promising paradigm for extreme length generalization, yet the key architectural principles underpinning its success are not yet fully understood. In this work, we present a systematic dissection of these models to identify the core components driving their performance. Through a unified framework and comprehensive ablation studies, we demonstrate that a combination of three design principles is critical: (1) an expressive, non-linear Chunk Encoder with a dedicated CLS token to produce representations for retrieval; (2) a Bypassing Residual Path to stably integrate retrieved global information without it being overridden by the local residual stream; and (3) enforced selection sparsity during pre-training to bridge the train-test distribution gap. We provide a theoretical motivation for intra-chunk information processing and landmark generation. By combining these principles, we establish a new state-of-the-art for training-free length extrapolation, successfully generalizing models trained on a 4K context to 32 million tokens on RULER and BABILong. Our findings provide a clear and empirically-grounded set of design principles for developing future, highly-capable long-context language models.