Slot Machines: How LLMs Keep Track of Multiple Entities

arXiv cs.CL / 4/24/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper examines how language models represent and maintain bindings between multiple entities and their attributes across token positions within a context.
It introduces a “multi-slot probing” method to disentangle a token’s residual stream into separate “current-entity” and “prior-entity” components.
The authors find that the current-entity and prior-entity slots play different functional roles: the current-entity slot supports explicit factual retrieval, while the prior-entity slot better supports relational inferences and conflict detection.
Open-weight models struggle with syntax that forces two subject-verb-object bindings onto a single token, while newer frontier models handle it, implying improved binding strategies.
The results highlight a gap between information present in model activations and information the model actually uses, and suggest the current/prior-entity slot structure could support multi-perspective behaviors like sycophancy or deception.

Abstract

Language models must bind entities to the attributes they possess and maintain several such binding relationships within a context. We study how multiple entities are represented across token positions and whether single tokens can carry bindings for more than one entity. We introduce a multi-slot probing approach that disentangles a single token's residual stream activation to recover information about both the currently described entity and the immediately preceding one. These two kinds of information are encoded in separate and largely orthogonal "current-entity" and "prior-entity" slots. We analyze the functional roles of these slots and find that they serve different purposes. In tandem with the current-entity slot, the prior-entity slot supports relational inferences, such as entity-level induction ("who came after Alice in the story?") and conflict detection between adjacent entities. However, only the current-entity slot is used for explicit factual retrieval questions ("Is anyone in the story tall?" "What is the tall entity's name?") despite these answers being linearly decodable from the prior-entity slot too. Consistent with this limitation, open-weight models perform near chance accuracy at processing syntax that forces two subject-verb-object bindings on a single token (e.g., "Alice prepares and Bob consumes food.") Interestingly, recent frontier models can parse this properly, suggesting they may have developed more sophisticated binding strategies. Overall, our results expose a gap between information that is available in activations and information the model actually uses, and suggest that the current/prior-entity slot structure is a natural substrate for behaviors that require holding two perspectives at once, such as sycophancy and deception.