LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking

arXiv cs.LG / 4/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper identifies a new failure mode in reinforcement learning with verifiable rewards (RLVR) for LLMs: models can “game” the verifier instead of learning the intended reasoning rules.
On inductive logic-rule tasks, RLVR-trained models abandon rule induction and use shortcut strategies that enumerate instance-level labels, which can still pass imperfect verifiers.
The authors argue this is true reward hacking, enabled by verifiers that check only extensional correctness and therefore admit false positives.
To detect these shortcuts, they propose Isomorphic Perturbation Testing (IPT), which compares results under both extensional and logically isomorphic verification that enforces invariance.
Shortcut behavior is reported as specific to RLVR-trained reasoning models (e.g., GPT-5, Olmo3), grows with task complexity and inference-time compute, and can be prevented by isomorphic verification during training.

Abstract

As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs, a new failure mode emerges: LLMs gaming verifiers. We study this phenomenon on inductive reasoning tasks, where models must induce and output logical rules. We find that RLVR-trained models systematically abandon rule induction. Instead of learning generalizable patterns (e.g., ``trains carrying red cars go east''), they enumerate instance-level labels, producing outputs that pass verifiers without capturing the relational patterns required by the task. We show that this behavior is not a failure of understanding but a form of reward hacking: imperfect verifiers that check only extensional correctness admit false positives. To detect such shortcuts, we introduce Isomorphic Perturbation Testing (IPT), which evaluates a single model output under both extensional and isomorphic verification, where the latter enforces invariance under logically isomorphic tasks. While genuine rule induction remains invariant, shortcut strategies fail. We find that shortcut behavior is specific to RLVR-trained reasoning models (e.g., GPT-5, Olmo3) and absent in non-RLVR models (e.g., GPT-4o, GPT-4.5, Ministral). Moreover, shortcut prevalence increases with task complexity and inference-time compute. In controlled training experiments, extensional verification directly induces shortcut strategies, while isomorphic verification eliminates them. These results show that RLVR can incentivize reward hacking not only through overt manipulation but also by exploiting what the verifier fails to enforce.