Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games

arXiv cs.AI / 5/7/2026

📰 NewsDeveloper Stack & InfrastructureIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that LLMs struggle in multi-agent games because outcomes depend on joint strategies, and the changing behavior of other agents makes evaluation and credit assignment across reasoning steps difficult.
It introduces Strat-Reasoner, an RL-based framework that boosts LLM strategic reasoning by using a recursive paradigm where an agent’s reasoning explicitly integrates other agents’ reasoning.
To better supervise intermediate reasoning, Strat-Reasoner uses a centralized Chain-of-Thought comparison module that evaluates the quality of reasoning sequences.
The method computes a hybrid advantage signal and applies a group-relative RL approach to optimize the LLM policy in multi-agent settings.
Experiments on multiple multi-agent games show an average 22.1% performance improvement over baseline LLM strategic abilities.

Abstract

While Large Language Models (LLMs) excel in certain reasoning tasks, they struggle in multi-agent games where the final outcome depends on the joint strategies of all agents. In multi-agent games, the non-stationarity of other agents brings significant challenges on the evaluation of the reasoning process and the credit assignment over multiple reasoning steps. Existing single-agent reinforcement learning (RL) approaches and their multi-agent extensions fail to address these challenges as they do not incorporate other agents in the reasoning process. In this work, we propose Strat-Reasoner, a novel RL-based framework that improves LLMs' strategic reasoning ability in multi-agent games. We introduce a novel recursive reasoning paradigm where an agent's reasoning also integrates other agents' reasoning processes. To provide effective reward signals for the intermediate reasoning sequences, we employ a centralized Chain-of-Thought (CoT) comparison module to evaluate the reasoning quality. Finally, we compute an accurate hybrid advantage and develop a group-relative RL approach to optimize the LLM policy. Experimental results show that Strat-Reasoner substantially improves strategic abilities of underlying LLMs, achieving 22.1\% average performance improvements across various multi-agent games.