Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation

arXiv cs.CL / 4/7/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper addresses the challenge of certifying LLM failure rates when human-labeled “gold” data is expensive and automatically labeled data (e.g., LLM-as-a-Judge) can be biased.
It proposes a constrained maximum-likelihood estimation framework that fuses three inputs: a small high-quality human calibration set, a large set of judge annotations, and domain-specific constraints based on known bounds of judge performance statistics.
Empirical results across multiple experimental conditions (judge accuracy, calibration size, and underlying LLM failure rate) show the constrained MLE approach achieves higher accuracy and lower variance than prior baselines such as Prediction-Powered Inference (PPI).
The authors emphasize that the method replaces “black-box” judge usage with a more interpretable, scalable, and principled pathway for LLM failure-rate certification.

Abstract

The ability to rigorously estimate the failure rates of large language models (LLMs) is a prerequisite for their safe deployment. Currently, however, practitioners often face a tradeoff between expensive human gold standards and potentially severely-biased automatic annotation schemes such as "LLM-as-a-Judge" labeling. In this paper, we propose a new, practical, and efficient approach to LLM failure rate estimation based on constrained maximum-likelihood estimation (MLE). Our method integrates three distinct signal sources: (i) a small, high-quality human-labeled calibration set, (ii) a large corpus of LLM-judge annotations, and, most importantly, (iii) additional side information via domain-specific constraints derived from known bounds on judge performance statistics. We validate our approach through a comprehensive empirical study, benchmarking it against state-of-the-art baselines like Prediction-Powered Inference (PPI). Across diverse experimental regimes -- spanning varying judge accuracies, calibration set sizes, and LLM failure rates -- our constrained MLE consistently delivers more accurate and lower-variance estimates than existing methods. By moving beyond the "black-box" use of automated judges to a flexible framework, we provide a principled, interpretable, and scalable pathway towards LLM failure-rate certification.