Meta-Harness: End-to-End Optimization of Model Harnesses

arXiv cs.AI / 3/31/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that LLM performance depends not only on model weights but also on the “harness” code that controls what context is stored, retrieved, and presented to the model.
It introduces Meta-Harness, an outer-loop agentic system that searches over harness code by using a proposer to access source code and evaluate candidates via scoring and execution traces recorded on the filesystem.
On online text classification, Meta-Harness improves accuracy over a state-of-the-art context management approach by 7.7 points while reducing context token usage by 4x.
For retrieval-augmented math reasoning, a single automatically discovered harness boosts accuracy by 4.7 points on average across five held-out models over 200 IMO-level problems.
In agentic coding tasks, the discovered harnesses outperform the best hand-engineered baselines on TerminalBench-2, suggesting automated harness engineering can materially improve real applications.

Abstract

The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.