What can LLMs tell us about the mechanisms behind polarity illusions in humans? Experiments across model scales and training steps

arXiv cs.CL / 3/31/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper uses the Pythia scaling suite to test whether the NPI and depth charge polarity illusions observed in humans also emerge in LLMs across model sizes and training steps.
It finds that the NPI illusion weakens and eventually disappears as model scale increases, while the depth charge illusion strengthens in larger models.
The authors argue that these patterns may reduce the need to posit human “rational inference” mechanisms that transform ill-formed sentences into well-formed ones, since LLMs cannot plausibly perform such implicit next-token reasoning.
Instead, the results suggest that shallow, “good enough” processing and/or partial grammaticalization of prescriptively ungrammatical structures could explain the illusions in both models and humans.
The study proposes a unifying theoretical synthesis grounded in construction grammar to relate these mechanisms across the different illusion types.

Abstract

I use the Pythia scaling suite (Biderman et al. 2023) to investigate if and how two well-known polarity illusions, the NPI illusion and the depth charge illusion, arise in LLMs. The NPI illusion becomes weaker and ultimately disappears as model size increases, while the depth charge illusion becomes stronger in larger models. The results have implications for human sentence processing: it may not be necessary to assume "rational inference" mechanisms that convert ill-formed sentences into well-formed ones to explain polarity illusions, given that LLMs cannot plausibly engage in this kind of reasoning, especially at the implicit level of next-token prediction. On the other hand, shallow, "good enough" processing and/or partial grammaticalization of prescriptively ungrammatical structures may both occur in LLMs. I propose a synthesis of different theoretical accounts that is rooted in the basic tenets of construction grammar.