The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious

arXiv cs.CL / 4/17/2026

📰 NewsDeveloper Stack & InfrastructureIdeas & Deep AnalysisModels & Research

共有:

Key Points

Turn-level evaluation metrics for multi-turn LLM conversations often assume statistical independence between consecutive turns, even though turns are autocorrelated.
An analysis of 66 turn-level metrics across 202 conversations shows that naive pooled testing can massively overstate significance, with 42% of seemingly significant associations failing after cluster-robust correction.
The magnitude of inflated findings varies by metric family: memoryless metric groups account for 14% of false positives, while non-memoryless groups account for 33%, with per-category rates ranging from 0% to 100%.
The paper proposes a two-stage correction framework (effective degrees of freedom via Chelton and a conversation-level block bootstrap) and reports improved replication on a pre-registered hold-out set.
A survey of ~30 recent NLP/AI papers finds that only 4 fully address temporal dependence for turn-level statistics, and 26 do not correct for it, indicating a widespread methodological gap.
categories_readable_note: "(Not included in output)"

Abstract

Turn-level metrics are widely used to evaluate properties of multi-turn human-LLM conversations, from safety and sycophancy to dialogue quality. However, consecutive turns within a conversation are not statistically independent -- a fact that virtually all current evaluation pipelines fail to correct for in their statistical inference. We systematically characterize the autocorrelation structure of 66 turn-level metrics across 202 multi-turn conversations (11,639 turn pairs, 5 German-speaking users, 4 LLM platforms) and demonstrate that naive pooled analysis produces severely inflated significance estimates: 42% of associations that appear significant under standard pooled testing fail to survive cluster-robust correction. The inflation varies substantially across categories rather than scaling linearly with autocorrelation: three memoryless families (embedding velocity, directional, differential) aggregate to 14%, while the seven non-memoryless families (thermo-cycle, frame distance, lexical/structural, rolling windows, cumulative, interaction, timestamp) aggregate to 33%, with individual category rates ranging from 0% to 100% depending on per-family effect size. We present a two-stage correction framework combining Chelton (1983) effective degrees of freedom with conversation-level block bootstrap, and validate it on a pre-registered hold-out split where cluster-robust metrics replicate at 57% versus 30% for pooled-only metrics. We provide concrete design principles, a publication checklist, and open-source code for the correction pipeline. A survey of ~30 recent papers at major NLP and AI venues that compute turn-level statistics in LLM evaluations finds that only 4 address temporal dependence at all, and 26 do not correct for it.