Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

arXiv cs.CL / 4/16/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper introduces Implicit Prefix-Value Reward Models (IPVRM) to improve Process Reward Models by learning prefix-conditioned value functions that estimate eventual correctness from trajectory-level outcome labels.
It addresses the train–inference mismatch of prior implicit reward approaches, which weakly identify token-level credits and can reinforce incorrect continuations due to miscalibration.
IPVRM derives token/step signals using temporal-difference (TD) differences, and the authors report substantial gains in step-verification F1 on ProcessBench.
Building on IPVRM’s calibrated prefix values, the paper proposes Distribution-Level RL (DistRL), which uses TD advantages for both sampled tokens and high-probability candidate tokens to enable dense counterfactual updates without extra rollouts.
DistRL shows limited gains when using miscalibrated implicit rewards, but consistently improves downstream reasoning when paired with IPVRM, highlighting the importance of reward calibration.

Abstract

Process reward models (PRMs) provide fine-grained reward signals along the reasoning process, but training reliable PRMs often requires step annotations or heavy verification pipelines, making them expensive to scale and refresh during online RL. Implicit PRMs mitigate this cost by learning decomposable token- or step-level rewards from trajectory-level outcome labels. However, they suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality. As a result, token-level credits are weakly identified and may fail to faithfully reflect which reasoning steps are actually correct. This unreliability undermines a key promise of implicit PRMs: scoring many candidate tokens. In practice, noisy per-token advantages may systematically reinforce incorrect continuations. We address this problem with a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness, and derives step signals via temporal-difference (TD) differences. IPVRM substantially improves step-verification F1 on ProcessBench. Building on these calibrated prefix values, we further propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens, enabling dense counterfactual updates without additional rollouts. While DistRL offers limited gains when powered by miscalibrated implicit rewards, it consistently improves downstream reasoning once paired with IPVRM.