Structured In-context Environment Scaling for Large Language Model Reasoning

arXiv cs.CL / 5/4/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that LLM reasoning improves via reinforcement learning (RL) environmental exploration, and that the environment’s intrinsic properties strongly constrain what models can learn.
It identifies limitations of existing environments: mathematical/coding setups often scale poorly due to reliance on expert annotations, while game-based environments tend to produce skills that don’t generalize.
The proposed Structured In-context Environment (SIE) framework automatically builds reasoning environments from large-scale structured data to achieve scalability and support compositional, generalizable reasoning.
SIE also targets verifiability by using explicit schemas and reasoning chains from structured data as a basis for rule-based checking.
Experiments indicate SIE improves in-domain structured reasoning and transfers learned skills to out-of-domain math and logic tasks, with additional gains even when learning from information-limited partial environments.
Point 5

Abstract

Large language models (LLMs) have achieved significant advancements in reasoning capabilities through reinforcement learning (RL) via environmental exploration. As the intrinsic properties of the environment determine the abilities that LLMs can learn, the environment plays a important role in the RL finetuning process. An ideal LLM reasoning environment should possess three core characteristics: scalability, generalizable reasoning, and verifiability. However, existing mathematical and coding environments are difficult to scale due to heavy reliance on expert annotation, while the skills learned in game-based environments are too specialized to generalize. To bridge this gap, we introduce the \textbf{S}tructured \textbf{I}n-context \textbf{E}nvironment (SIE) framework. SIE achieves scalability by automatically constructing reasoning environments from large-scale structured data, where the rich compositional patterns naturally support generalizable reasoning. Moreover, the explicit schemas and reasoning chains in structured data provide a foundation for rule-based verifiability. Experimental results show that SIE framework not only achieves substantial improvements in in-domain structured reasoning, but also enables the learned compositional reasoning skills to generalize effectively to out-of-domain mathematical and logical reasoning tasks. We further explored learning in information-limited partial SIEs and found that LLMs can infer the missing information through exploring the environment, leading to robust reasoning improvements and generalization performance.