AI Navigate
インサイト最新記事一覧AI大全

AI benchmarks systematically ignore how humans disagree, Google study finds

THE DECODER / 4/5/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research
Read original →
共有:

Key Points

  • A Google study argues that common AI benchmark practices using only three to five human raters per example can produce unreliable results because it fails to capture variability in human judgment.
  • The research finds that how teams split their annotation budget across items and raters can matter as much as the total number of annotations collected.
  • The study highlights that benchmark scores may be systematically biased when human disagreement is treated as noise rather than an informative signal.
  • It implies that future benchmark design should account for rater disagreement and uncertainty to improve comparability and robustness across models.

Coloured contour and dot patterns are superimposed on a faceless human bust and symbolize data visualization of human benchmarks.

A Google study finds that the standard three to five human raters per test example often aren't enough for reliable AI benchmarks, and that splitting your annotation budget the right way matters just as much as the budget itself.

The article AI benchmarks systematically ignore how humans disagree, Google study finds appeared first on The Decoder.

💡 Insights using this article

This article is featured in our daily AI news digest — key takeaways and action items at a glance.

📅 4/5DailyView insight →

関連おすすめサービス

※当サイトはアフィリエイト広告を利用しています
🎧

Notta搭載AI議事録イヤホン ZENCHORD1

AI時代の仕事術。Notta搭載で会議の議事録を自動生成するスマートイヤホン。

🎙️

AI搭載ボイスレコーダー Plaud

世界100万人が愛用。AIで文字起こし・要約を自動化するボイスレコーダー。

🖼️

画像高画質化AIツール Aiarty Image Enhancer

AIで画像を高画質化。写真・イラストを簡単にアップスケール。