Bias at the End of the Score

arXiv cs.CV / 4/16/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

Reward models (RMs) are described as inherently non-neutral functions that are widely used across text-to-image pipelines for filtering, evaluation, optimization guidance, and safety/quality scoring.
The study performs a large-scale audit of RM robustness and finds that, beyond quality measurement, RMs encode demographic biases.
The authors report that reward-guided optimization can sexualize female image subjects, reinforce gender and racial stereotypes, and reduce demographic diversity.
The findings suggest that current RMs are not reliably fair or robust as scoring functions, undermining their usefulness as quality metrics in T2I systems.
The paper calls for improved data collection and training procedures to build reward models that provide more robust and equitable scoring during generation.

Abstract

Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring.