BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

arXiv cs.CL / 4/13/2026

💬 OpinionIdeas & Deep AnalysisTools & Practical UsageModels & Research

共有:

Key Points

The paper argues that conventional reference-based LLM evaluation often depends on brittle lexical methods that can mis-measure true reasoning by overemphasizing compliance with rigid formatting rules.
A large empirical study across 36 models and 15 tasks finds lexical evaluation correlates poorly with human judgments, motivating a more semantic approach.
It introduces BERT-as-a-Judge, an encoder-based evaluator trained (lightweight) on synthetically annotated question–candidate–reference triplets to assess answer correctness robustly despite paraphrasing.
The authors report that BERT-as-a-Judge beats lexical baselines while matching the quality of much larger LLM judge systems, offering a favorable compute-to-accuracy tradeoff.
The work includes extensive analysis and releases artifacts to help practitioners adopt the method for scalable, reliable LLM evaluation.

Abstract

Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling tradeoff between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.