Reflective Context Learning: Studying the Optimization Primitives of Context Space

arXiv cs.LG / 4/6/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

This paper argues that many core learning optimization challenges (credit assignment, overfitting/forgetting, local optima, high-variance signals) also arise when learning is done in context space rather than parameter space, and that current methods are fragmented.
It introduces Reflective Context Learning (RCL), a unified agent framework where reflection produces a gradient-like directional update signal from trajectories and current context, and mutation applies it to iteratively improve future context.
The authors reinterpret prior context-optimization approaches as special cases of a shared learning-and-optimization problem and extend the framework with reusable optimization primitives such as batching, better credit-assignment signals, auxiliary losses, failure replay, and grouped rollouts for variance reduction.
Experiments on AppWorld, BrowseComp+, and RewardBench2 show that these primitives improve performance over strong baselines, with their relative value changing across different task regimes.
The study further analyzes how design choices (initialization robustness, batch size, sampling/curriculum, optimizer-state variants, and allocating different model strengths across components) affect outcomes, supporting the view that context-updating should be treated as a systematic optimization problem.

Abstract

Generally capable agents must learn from experience in ways that generalize across tasks and environments. The fundamental problems of learning, including credit assignment, overfitting, forgetting, local optima, and high-variance learning signals, persist whether the learned object lies in parameter space or context space. While these challenges are well understood in classical machine learning optimization, they remain underexplored in context space, leading current methods to be fragmented and ad hoc. We present Reflective Context Learning (RCL), a unified framework for agents that learn through repeated interaction, reflection on behavior and failure modes, and iterative updates to context. In RCL, reflection converts trajectories and current context into a directional update signal analogous to gradients, while mutation applies that signal to improve future behavior in context space. We recast recent context-optimization approaches as instances of this shared learning problem and systematically extend them with classical optimization primitives, including batching, improved credit-assignment signal, auxiliary losses, failure replay, and grouped rollouts for variance reduction. On AppWorld, BrowseComp+, and RewardBench2, these primitives improve over strong baselines, with their relative importance shifting across task regimes. We further analyze robustness to initialization, the effects of batch size, sampling and curriculum strategy, optimizer-state variants, and the impact of allocating stronger or weaker models to different optimization components. Our results suggest that learning through context updates should be treated not as a set of isolated algorithms, but as an optimization problem whose mechanisms can be studied systematically and improved through transferable principles.