Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoring

arXiv cs.AI / 4/22/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper introduces a scalable experimental framework to measure how sensitive LLM-based “judges” are to small semantic changes when comparing two documents, varying many perturbation and context factors.
Testing five LLMs across tens of thousands of document pairs shows a positional bias: many models penalize semantic differences more strongly when they occur earlier in a document.
The study finds that putting a semantically altered sentence in topically unrelated surrounding context lowers similarity scores and can produce “bipolarized” outcomes (either very low or very high similarity).
Scoring behavior differs by model identity, yielding distinct but stable “fingerprints” for each LLM, while all models follow a shared ordering in how leniently they respond to different perturbation types.
The authors argue these results highlight that LLM similarity scoring depends not only on the semantic change itself, but also on document structure and context coherence, and they provide an LLM-agnostic auditing toolkit.

Abstract

We propose a scalable, multifactorial experimental framework that systematically probes LLM sensitivity to subtle semantic changes in pairwise document comparison. We analogize this as a needle-in-a-haystack problem: a single semantically altered sentence (the needle) is embedded within surrounding context (the hay), and we vary the perturbation type (negation, conjunction swap, named entity replacement), context type (original vs. topically unrelated), needle position, and document length across all combinations, testing five LLMs on tens of thousands of document pairs. Our analysis reveals several striking findings. First, LLMs exhibit a within-document positional bias distinct from previously studied candidate-order effects: most models penalize semantic differences more harshly when they occur earlier in a document. Second, when the altered sentence is surrounded by topically unrelated context, it systematically lowers similarity scores and induces bipolarized scores that indicate either very low or very high similarity. This is consistent with an interpretive frame account in which topically-related context may allow models to contextualize and downweight the alterations. Third, each LLM produces a qualitatively distinct scoring distribution, a stable "fingerprint" that is invariant to perturbation type, yet all models share a universal hierarchy in how leniently they treat different perturbation types. Together, these results demonstrate that LLM semantic similarity scores are sensitive to document structure, context coherence, and model identity in ways that go beyond the semantic change itself, and that the proposed framework offers a practical, LLM-agnostic toolkit for auditing and comparing scoring behavior across current and future models.