Exploration and Exploitation Errors Are Measurable for Language Model Agents

arXiv cs.AI / 4/16/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper proposes a policy-agnostic way to measure exploration versus exploitation errors in language model (LM) agents, even when the agent’s internal policy is not accessible.
It introduces controllable partially observable 2D grid environments with unknown task DAGs, where the difficulty can be tuned to emphasize exploration or exploitation.
The authors define a metric that infers exploration/exploitation errors from observed actions, enabling systematic evaluation across different LM agent approaches.
Experiments on multiple frontier LM agents show that state-of-the-art models still struggle, with notable differences in failure modes between models.
The study finds that reasoning-focused models perform better, and that both exploration and exploitation can be improved with relatively small harness (evaluation setup) engineering changes, alongside releasing the code.

Abstract

Language Model (LM) agents are increasingly used in complex open-ended decision-making tasks, from AI coding to physical AI. A core requirement in these settings is the ability to both explore the problem space and exploit acquired knowledge effectively. However, systematically distinguishing and quantifying exploration and exploitation from observed actions without access to the agent's internal policy remains challenging. To address this, we design controllable environments inspired by practical embodied AI scenarios. Each environment consists of a partially observable 2D grid map and an unknown task Directed Acyclic Graph (DAG). The map generation can be programmatically adjusted to emphasize exploration or exploitation difficulty. To enable policy-agnostic evaluation, we design a metric to quantify exploration and exploitation errors from agent's actions. We evaluate a variety of frontier LM agents and find that even state-of-the-art models struggle on our task, with different models exhibiting distinct failure modes. We further observe that reasoning models solve the task more effectively and show both exploration and exploitation can be significantly improved through minimal harness engineering. We release our code \href{https://github.com/jjj-madison/measurable-explore-exploit}{here}.