BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation

arXiv cs.CL / 4/6/2026

💬 OpinionIdeas & Deep AnalysisTools & Practical UsageModels & Research

共有:

Key Points

Web-enabled scientific publishing agents using large language models can generate BibTeX citations with widespread field-level errors, even though prior evaluations often ignored the role of web search.
The authors introduce a benchmark of 931 papers across four domains and multiple citation tiers, along with version-aware ground truth, and evaluate three frontier search-enabled models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) using a nine-field scoring scheme and a six-way error taxonomy.
While overall accuracy reaches 83.6%, only 50.9% of generated BibTeX entries are completely correct, and accuracy drops substantially for more recent papers, indicating strong reliance on parametric memory.
Two main failure modes are identified—wholesale entry substitution (identity fields failing together) and isolated field errors—supported by field-error co-occurrence analysis.
As mitigation, the paper evaluates clibib, an open-source deterministic BibTeX retrieval tool (Zotero Translation Server with CrossRef fallback), and shows that a two-stage “search then revision against authoritative records” integration increases accuracy to 91.5% with low regression (0.8%), outperforming a single-stage approach.

Abstract

Large language models with web search are increasingly used in scientific publishing agents, yet they still produce BibTeX entries with pervasive field-level errors. Prior evaluations tested base models without search, which does not reflect current practice. We construct a benchmark of 931 papers across four scientific domains and three citation tiers -- popular, low-citation, and recent post-cutoff -- designed to disentangle parametric memory from search dependence, with version-aware ground truth accounting for multiple citable versions of the same paper. Three search-enabled frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate BibTeX entries scored on nine fields and a six-way error taxonomy, producing ~23,000 field-level observations. Overall accuracy is 83.6%, but only 50.9% of entries are fully correct; accuracy drops 27.7pp from popular to recent papers, revealing heavy reliance on parametric memory even when search is available. Field-error co-occurrence analysis identifies two failure modes: wholesale entry substitution (identity fields fail together) and isolated field error. We evaluate clibib, an open-source tool for deterministic BibTeX retrieval from the Zotero Translation Server with CrossRef fallback, as a mitigation mechanism. In a two-stage integration where baseline entries are revised against authoritative records, accuracy rises +8.0pp to 91.5%, fully correct entries rise from 50.9% to 78.3%, and regression rate is only 0.8%. An ablation comparing single-stage and two-stage integration shows that separating search from revision yields larger gains and lower regression (0.8% vs. 4.8%), demonstrating that integration architecture matters independently of model capability. We release the benchmark, error taxonomy, and clibib tool to support evaluation and mitigation of citation hallucinations in LLM-based scientific writing.