Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?

arXiv cs.CL / 4/17/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies whether language models develop transferable representations for filler-gap dependencies across syntactic constructions such as wh-questions and topicalization.
It uses Distributed Alignment Search (DAS) on LMs trained with different amounts of data from the BabyLM challenge to probe how learning behaves under data-quantity constraints.
The findings indicate that shared—but item-sensitive—mechanisms can emerge even with limited training data.
However, the models still need substantially more data than humans to reach generalizations comparable to human language acquisition, suggesting a role for language-specific inductive biases in acquisition theories.

Abstract

For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.