Decoupling Scores and Text: The Politeness Principle in Peer Review

arXiv cs.LG / 4/17/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies why authors misread peer review by comparing predictive power from numerical scores versus free-text reviews across 30,000+ ICLR submissions (2021–2025).
It finds a clear accuracy gap: score-based models reach about 91% accuracy, while text-based models achieve around 81% even when using large language models, showing text is less reliable.
For cases where score-based models fail, the authors observe score distributions with high kurtosis and negative skewness, suggesting that extreme low scores (not the mean) heavily drive rejection.
From a sentiment perspective, the authors attribute the weaker text signal to the Politeness Principle: reviews of rejected papers tend to include more positive than negative sentiment terms, which can obscure the rejection cue.

Abstract

Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores. To investigate this, we construct a dataset of over 30,000 ICLR 2021-2025 submissions and compare acceptance prediction performance using numerical scores versus text reviews. Our experiments reveal a significant performance gap: score-based models achieve 91% accuracy, while text-based models reach only 81% even with large language models, indicating that textual information is considerably less reliable. To explain this phenomenon, we first analyze the 9% of samples that score-based models fail to predict, finding their score distributions exhibit high kurtosis and negative skewness, which suggests that individual low scores play a decisive role in rejection even when the average score falls near the borderline. We then examine why text-based accuracy significantly lags behind scores from a review sentiment perspective, revealing the prevalence of the Politeness Principle: reviews of rejected papers still contain more positive than negative sentiment words, masking the true rejection signal and making it difficult for authors to judge outcomes from text alone.