Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance

arXiv cs.AI / 3/31/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that task-completion rate alone can miss important differences in how well LLM agents track intermediate cumulative state.
It introduces WMF-AM, a “no-scratchpad” calibrated probe for cumulative arithmetic state tracking, and evaluates it across 20 open-weight model families.
In a pre-specified multiple-comparison-corrected analysis, WMF-AM significantly predicts deterministic 10-task agent performance (Kendall’s tau = 0.612, p < 0.001).
Construct-isolation ablations indicate that the main challenge for agents under load is cumulative state tracking, not just single-step arithmetic or entity tracking.
The authors note that K-calibration helps keep the probe discriminative versus earlier fixed-depth benchmarks, while generalization beyond the studied open-weight set remains an open question.

Abstract

Task-completion rate is the standard proxy for LLM agent capability, but models with identical completion scores can differ substantially in their ability to track intermediate state. We introduce Working Memory Fidelity-Active Manipulation (WMF-AM), a calibrated no-scratchpad probe of cumulative arithmetic state tracking, and evaluate it on 20 open-weight models (0.5B-35B, 13 families) against a released deterministic 10-task agent battery. In a pre-specified, Bonferroni-corrected analysis, WMF-AM predicts agent performance with Kendall's tau = 0.612 (p < 0.001, 95% CI [0.360, 0.814]); exploratory partial-tau analyses suggest this signal persists after controlling for completion score and model scale. Three construct-isolation ablations (K = 1 control, non-arithmetic ceiling, yoked cancellation) support the interpretation that cumulative state tracking under load, rather than single-step arithmetic or entity tracking alone, is the primary difficulty source. K-calibration keeps the probe in a discriminative range where prior fixed-depth benchmarks become non-discriminative; generalization beyond this open-weight sample remains open.