Cell-Based Representation of Relational Binding in Language Models

arXiv cs.CL / 4/22/2026

📰 NewsModels & Research

共有:

Key Points

The paper investigates how LLMs perform discourse-level relational binding, a process needed to track entities and the relations between them over multiple sentences.
It proposes and tests a “Cell-based Binding Representation (CBR),” where LLMs encode entity–relation index pairs as cells within a low-dimensional linear subspace and retrieve bound attributes from the relevant cell during inference.
Using controlled multi-sentence datasets with entity and relation indices, the authors identify the CBR subspace via Partial Least Squares regression and show the indices are linearly decodable across multiple domains and two model families.
The authors find a grid-like geometry in the learned representation space and show that context-specific CBRs are connected by translation vectors, enabling cross-context transfer.
Activation patching and targeted perturbations provide causal evidence that manipulating the CBR subspace changes relational predictions and can disrupt model performance.

Abstract

Understanding a discourse requires tracking entities and the relations that hold between them. While Large Language Models (LLMs) perform well on relational reasoning, the mechanism by which they bind entities, relations, and attributes remains unclear. We study discourse-level relational binding and show that LLMs encode it via a Cell-based Binding Representation (CBR): a low-dimensional linear subspace in which each ``cell'' corresponds to an entity--relation index pair, and bound attributes are retrieved from the corresponding cell during inference. Using controlled multi-sentence data annotated with entity and relation indices, we identify the CBR subspace by decoding these indices from attribute-token activations with Partial Least Squares regression. Across domains and two model families, the indices are linearly decodable and form a grid-like geometry in the projected space. We further find that context-specific CBR representations are related by translation vectors in activation space, enabling cross-context transfer. Finally, activation patching shows that manipulating this subspace systematically changes relational predictions and that perturbing it disrupts performance, providing causal evidence that LLMs rely on CBR for relational binding.