AI Navigate
インサイト最新記事AI大全カオスマップ

ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents

arXiv cs.CL / 4/10/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research
Read original →
共有:

Key Points

  • The paper introduces Oracle-SWE, a unified approach for isolating and extracting key “oracle” information signals (e.g., reproduction/regression tests, edit location, execution context, and API usage) from SWE benchmarks to measure their individual effect on success.
  • It targets a gap in prior research by quantifying how much each signal contributes when the intermediate information is assumed to be perfectly available, rather than only studying end-to-end agent performance.
  • The study further tests whether signals produced by strong language models can be used to approximate real-world settings by feeding extracted signals into a base SWE agent and measuring performance gains.
  • The findings are intended to help guide research prioritization for autonomous coding/agentic software engineering systems by clarifying which contextual signals matter most.
  • Overall, the work reframes SWE-agent evaluation as a controllable, signal-level ablation/attribution problem to better understand what drives agent improvements.

Abstract

Recent advances in language model (LM) agents have significantly improved automated software engineering (SWE). Prior work has proposed various agentic workflows and training strategies as well as analyzed failure modes of agentic systems on SWE tasks, focusing on several contextual information signals: Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. However, the individual contribution of each signal to overall success remains underexplored, particularly their ideal contribution when intermediate information is perfectly obtained. To address this gap, we introduce Oracle-SWE, a unified method to isolate and extract oracle information signals from SWE benchmarks and quantify the impact of each signal on agent performance. To further validate the pattern, we evaluate the performance gain of signals extracted by strong LMs when provided to a base agent, approximating real-world task-resolution settings. These evaluations aim to guide research prioritization for autonomous coding systems.

関連おすすめサービス

※当サイトはアフィリエイト広告を利用しています
🎧

Notta搭載AI議事録イヤホン ZENCHORD1

AI時代の仕事術。Notta搭載で会議の議事録を自動生成するスマートイヤホン。

🎙️

AI搭載ボイスレコーダー Plaud

世界100万人が愛用。AIで文字起こし・要約を自動化するボイスレコーダー。

🖼️

画像高画質化AIツール Aiarty Image Enhancer

AIで画像を高画質化。写真・イラストを簡単にアップスケール。