Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

arXiv cs.LG / 4/6/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper highlights that reward models used in RLHF are vulnerable to reward hacking, with prior attacks largely manipulating outputs in the semantic (human-readable) text space.
It introduces Token Mapping Perturbation Attack (TOMPA), which performs adversarial optimization directly in token space to bypass the usual decode→re-tokenize step between policy and reward model.
TOMPA uses only black-box scalar reward feedback to automatically find non-linguistic token patterns that trigger very high RM scores across multiple state-of-the-art reward models.
When targeting Skywork-Reward-V2-Llama-3.1-8B, TOMPA nearly doubles the reward of GPT-5 reference answers and exceeds them on 98% of prompts, while producing degenerate nonsensical text.
The results suggest a critical vulnerability in current RLHF pipelines: reward models can be systematically exploited beyond the semantic regime, indicating limitations of semantic-only defenses.

Abstract

Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF), yet they remain vulnerable to reward hacking. Existing attacks mainly operate within the semantic space, constructing human-readable adversarial outputs that exploit RM biases. In this work, we introduce a fundamentally different paradigm: Token Mapping Perturbation Attack (TOMPA), a framework that performs adversarial optimization directly in token space. By bypassing the standard decode-re-tokenize interface between the policy and the reward model, TOMPA enables the attack policy to optimize over raw token sequences rather than coherent natural language. Using only black-box scalar feedback, TOMPA automatically discovers non-linguistic token patterns that elicit extremely high rewards across multiple state-of-the-art RMs. Specifically, when targeting Skywork-Reward-V2-Llama-3.1-8B, TOMPA nearly doubles the reward of GPT-5 reference answers and outperforms them on 98.0% of prompts. Despite these high scores, the generated outputs degenerate into nonsensical text, revealing that RMs can be systematically exploited beyond the semantic regime and exposing a critical vulnerability in current RLHF pipelines.