Lost in State Space: Probing Frozen Mamba Representations

arXiv cs.CL / 5/4/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper tests a hypothesis that Mamba’s recurrent state h_t can be used to extract token-level outputs at fixed patch boundaries to obtain semantic sentence representations without pooling heads or fine-tuning.
Experiments on five benchmarks (SST-2, CoLA, MRPC, STS-B, IMDb) show that patch-boundary readouts from a frozen Mamba-130M backbone do not reliably beat simple mean pooling.
The authors find major representation issues, including extremely high anisotropy and representational collapse in the raw final SSM state (e.g., MCC = 0.000 on CoLA across multiple seeds).
They propose “orthogonal injection,” a modified recurrence intended to constrain how new information is incorporated, addressing the identified structural pathologies.

Abstract

Mamba's recurrent state h_t is, by construction, a compressed summary of every token seen so far. This raises a tempting hypothesis: if we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free, with no pooling head, no fine-tuning, and no [CLS] token. We test this hypothesis carefully. Across five benchmarks (SST-2, CoLA, MRPC, STS-B, IMDb), we compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone under a strict frozen-feature probing protocol, using three random seeds where computationally feasible. The results do not support the hypothesis: patch boundary readouts do not consistently outperform simple mean pooling. We identify and quantify two structural pathologies: severe anisotropy (mean pairwise cosine similarity 0.9999, std 0.000044) and representational collapse in the raw final SSM state (MCC = 0.000 on CoLA across all three seeds, confirmed via confusion matrix). We further propose orthogonal injection, a modified recurrence that constrains new information per