AI Navigate

[D] What is even the point of these LLM benchmarking papers?

Reddit r/MachineLearning / 3/13/2026

💬 OpinionSignals & Early TrendsIdeas & Deep Analysis
Read original →
共有:

Key Points

  • LLM benchmarking papers have proliferated at major conferences, but their usefulness is questioned because they benchmark proprietary models that change rapidly.
  • Proprietary LLMs are updated almost every month, and older versions can be deprecated or disappear, making results outdated by the time of publication.
  • The post asks whether big tech companies actually use these benchmark results to improve their models, highlighting a potential gap between benchmarks and real-world impact.
  • Suggestions include building dynamic, continuous evaluation benchmarks, open and reproducible suites, and time-aware leaderboards that track model performance over successive releases.

Lately, NeurIPS and ICLR are flooded with these LLM benchmarking papers. All they do is take a problem X and benchmark a bunch of propriety LLMs on this problem. My main question is these proprietary LLMs are updated almost every month. The previous models are deprecated and are sometimes no longer available. By the time these papers are published, the models they benchmark on are already dead.

So, what is the point of such papers? Are these big tech companies actually using the results from these papers to improve their models?

submitted by /u/casualcreak
[link] [comments]

関連おすすめサービス

※当サイトはアフィリエイト広告を利用しています
🎧

Notta搭載AI議事録イヤホン ZENCHORD1

AI時代の仕事術。Notta搭載で会議の議事録を自動生成するスマートイヤホン。

🎙️

AI搭載ボイスレコーダー Plaud

世界100万人が愛用。AIで文字起こし・要約を自動化するボイスレコーダー。

🖼️

画像高画質化AIツール Aiarty Image Enhancer

AIで画像を高画質化。写真・イラストを簡単にアップスケール。