Long-form RewardBench: Evaluating Reward Models for Long-form Generation

arXiv cs.CL / 3/16/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

Long-form RewardBench is introduced as the first benchmark specifically designed to evaluate reward models for long-form generation, covering sub-tasks such as QA, RAG, Chat, Writing, and Reasoning.
The authors collected instruction and preference data through a multi-stage process and evaluated 20+ reward models, including both classifiers and generative models.
Findings show current reward models struggle with long-form reward modeling, including a Long-form Needle-in-a-Haystack Test that links performance to error position and response length, with distinct behaviors between classification and generative models, and classifiers generalizing better than generative models trained on the same data.
As the first benchmark of its kind, Long-form RewardBench aims to provide a robust platform for visualizing progress in long-form reward modeling.

Abstract

The widespread adoption of reinforcement learning-based alignment highlights the growing importance of reward models. Various benchmarks have been built to evaluate reward models in various domains and scenarios. However, a significant gap remains in assessing reward models for long-form generation, despite its critical role in real-world applications. To bridge this, we introduce Long-form RewardBench, the first reward modeling testbed specifically designed for long-form generation. Our benchmark encompasses five key subtasks: QA, RAG, Chat, Writing, and Reasoning. We collected instruction and preference data through a meticulously designed multi-stage data collection process, and conducted extensive experiments on 20+ mainstream reward models, including both classifiers and generative models. Our findings reveal that current models still lack long-form reward modeling capabilities. Furthermore, we designed a novel Long-form Needle-in-a-Haystack Test, which revealed a correlation between reward modeling performance and the error's position within a response, as well as the overall response length, with distinct characteristics observed between classification and generative models. Finally, we demonstrate that classifiers exhibit better generalizability compared to generative models trained on the same data. As the first benchmark for long-form reward modeling, this work aims to offer a robust platform for visualizing progress in this crucial area.