LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text

arXiv cs.CL / 4/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The study used GPT-4.1 to infer MLB fans’ overall 0–10 experience ratings from a single open-ended text response, without any structured survey fields.
Across about 10,000 responses from five MLB teams, 67% of AI-predicted ratings were within ±1 point of the survey ratings, with 36% matching exactly.
The predictions were highly consistent across three independent scoring runs (87% exact agreement; 99.9% within ±1), but they were systematically about one point lower than self-reported ratings.
The model aligned most strongly with the fans’ overall evaluative verdict (correlation r = 0.82) rather than with individual components like parking, concessions, or staff.
The authors argue that the gap between predicted and self-reported scores reflects a meaningful “construct difference” (overall verdict vs. salient-moment impact), suggesting it may be useful to preserve rather than eliminate.

Abstract

We tasked GPT-4.1 to read what baseball fans wrote about their game-day experience and predict the overall experience rating each fan gave on a 0-10 survey scale. The model received only the text of a single open-ended response. These AI predictions were compared with the actual experience ratings captured by the survey instrument across approximately 10,000 fan responses from five Major League Baseball teams. In total two-thirds of predicted ratings fell within one point of self-reported fan ratings (67% within +/-1, 36% exact match), and the predicted measurement was near-deterministic across three independent scoring runs (87% exact agreement, 99.9% within +/-1). Predicted ratings aligned most strongly with the overall experience rating (r = 0.82) rather than with any specific aspect of the game-day experience such as parking, concessions, staff, etc. However, predictions were systematically lower than self-reported ratings by approximately one point, and this gap was not driven by any single aspect. Rather, our analysis shows that self-reported ratings capture the fan's verdict, an overall evaluative judgment that integrates the entire experience. While predicted ratings quantify the impact of salient moments characterized as memorable, emotionally intense, unusual, or actionable. Each measure contains information the other misses. These baseline results establish that a simple, unoptimized prompt can directionally predict how fans rate their experience from the text a fan wrote and that a gap between the two numbers can be interpreted as a construct difference worth preserving rather than an error to eliminate.