Dynamic Dual-Granularity Skill Bank for Agentic RL

arXiv cs.AI / 3/31/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper introduces D2Skill, a dynamic dual-granularity skill bank for agentic reinforcement learning that separates reusable experience into task-level skills for guidance and step-level skills for fine-grained decisions and error correction.
D2Skill jointly trains the policy and the skill bank using paired baseline and skill-injected rollouts, deriving hindsight utility signals from performance gaps to update both skills and policy optimization.
The skill bank is built solely from training-time experience and is continuously expanded and maintained via reflection, with utility-aware retrieval and pruning to keep the memory effective and current.
Experiments on ALFWorld and WebShop using Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 report consistent success-rate improvements of 10–20 points versus skill-free baselines.
Ablations indicate that both the dual-granularity modeling and dynamic skill maintenance are essential for the gains, and that the learned skills show higher utility, better transfer across evaluation settings, and only modest overhead.

Abstract

Agentic reinforcement learning (RL) can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld and WebShop with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 show that D2Skill consistently improves success rates over skill-free baselines by 10-20 points. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.