CRISP: Characterizing Relative Impact of Scholarly Publications

arXiv cs.CL / 3/31/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper introduces CRISP, a method that uses LLMs to jointly rank all cited works inside a citing paper to enable relative impact comparisons rather than evaluating citations in isolation.
To reduce LLM positional bias, CRISP repeats the ranking three times with randomized orderings and aggregates results using majority voting.
CRISP improves over a prior state-of-the-art impact classifier, achieving +9.5% accuracy and +8.3% F1 on a human-annotated citation dataset.
The approach is designed to be more efficient by requiring fewer LLM calls and can run competitively with an open-source model, supporting scalable and cost-effective analysis.
The authors release the produced rankings, impact labels, and a codebase to encourage follow-on research on citation impact characterization.

Abstract

Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose CRISP, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. CRISP outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. CRISP further gains efficiency through fewer LLM calls and performs competitively with an open-source model, enabling scalable, cost-effective citation impact analysis. We release our rankings, impact labels, and codebase to support future research.