Originally published on CoreProse KB-incidents
In 2026, more than fifty accepted ICLR papers were found to contain hallucinated citations, non‑existent datasets, and synthetic “results” generated by large language models—yet they passed peer review.[1][3] This reflected a systemic failure: generative AI was used without verification discipline in a high‑stakes publication pipeline.[1][3]
Similar failures have appeared in law, security, and software: fluent AI output was treated as truth while governance lagged.[1][2][10]
💼 Anecdote
A program chair at a smaller ML venue reported a “polished, clearly LLM‑written paper” that initially passed two overloaded reviewers—until a volunteer noticed that half the references resolved to nothing.[2] ICLR 2026 scaled up that same dynamic.
1. From Legal Sanctions to ICLR 2026: Integrity Problem, Not a Bug
Legal practice has already seen the “ChatGPT cites fake cases” phase.[1] In Mata v. Avianca and similar cases, judges sanctioned attorneys who submitted filings with hallucinated authorities, despite claims of ignorance about model limits.[1][4]
Studies of legal drafting tools show that even retrieval‑augmented systems fabricate citations for up to one‑third of complex queries.[2] These are commercial products, not prototypes.[2]
James’s taxonomy distinguishes:[1]
- Misgrounded errors: misquoting or misinterpreting real sources.
- Fully fabricated content: invented cases, statutes, or quotations.
ICLR 2026 mirrored this split:
- Misdescribed prior work (baselines, limitations).
- Cited non‑existent datasets, benchmarks, or “prior work” unreachable by any index.[1][2]
⚠️ Key point
Hallucinations are inherent to models optimizing next‑token likelihood, not truth.[1][3] Expecting the “next model” to fix this by default is unrealistic.
Legal scholars now frame hallucination‑driven errors as breaches of professional duty.[1][2] Shamov argues individual liability is insufficient given empirically unreliable “certified” tools, and proposes distributed liability across:[4]
- Tool developers
- Institutions and courts
- Practitioners
Conference publishing fits the same pattern:
- Vendors build writing and literature tools.
- Institutions and venues set policy and review processes.
- Authors and reviewers choose and validate outputs.
An integrity‑first workflow for AI‑heavy research should resemble legal and safety‑critical processes: multi‑layer hallucination mitigation, provenance logging, and disciplined human review.[2][3]
2. How Hallucinations Evade Peer Review: Technical Failure Modes in AI‑Assisted Writing
LLMs hallucinate because they generate plausible continuations under uncertainty, not verified facts.[1][3][8] Prompts like “summarize related work on X” or “suggest ablations” invite confident but possibly false text.
Common research‑paper hallucinations:[1][2]
- Fictitious references and venues.
- Non‑existent benchmarks/datasets with realistic names.
- Synthetic ablations never executed.
- Fabricated user studies with invented N and scores.
Legal filings show the same: fake cases in correct citation format.[1][2]
Hiriyanna and Zhao’s multi‑layer view clarifies the ICLR failures:[3]
- Data layer: unverified bibliographies; incomplete experiment metadata.
- Model layer: unconstrained, non‑deterministic generation for high‑stakes sections.
- Retrieval layer: weak grounding; vague prompts like “add more baselines.”
- Human layer: time‑pressed authors and reviewers, biased toward trusting fluent text.[3][8]
📊 Automation bias by analogy
With AI code assistants, 30–50% of generated snippets contain vulnerabilities, yet developers over‑trust them and reduce manual review.[10] Researchers under deadline, skimming LLM‑generated related work that “sounds right,” face the same risk.
Peer review remains mostly AI‑agnostic:
- No required provenance logs (which text used model X).
- No integrated citation resolvers or dataset registries.
- No checklists for AI‑induced risks.[2][6]
⚡ Pipeline sketch
Typical AI‑assisted paper pipeline in 2026:
- Prompt: “Draft related work on retrieval‑augmented generation for code search.”
- Drafting: LLM outputs polished text and ~10 citations.
- Light editing: authors tweak style; add a few real references.
- Submission: PDF uploaded; no AI‑usage or prompt record.
- Review: reviewers focus on novelty and experiments; they rarely verify every citation.
Hallucinations usually enter at step 2, survive step 3, and pass step 5, where they look like routine sloppiness rather than synthetic fabrication.[1][3][8]
3. Governance Lessons from Law, Security, and AI Platforms
Legal‑ethics proposals stress mandatory AI literacy, provenance logging, and human‑in‑the‑loop verification for any AI‑drafted filing.[2] Conferences can mirror this:
- AI literacy → author/reviewer training on hallucination risks.
- Provenance logging → AI‑usage disclosure in submissions.
- Human verification → explicit responsibilities per section.
Shamov’s distributed liability model suggests shared accountability among:[4]
- Tool vendors (minimum verification features, certification).
- Publishers and conferences (policies, audits, sanctions).
- Professionals (duty to verify and disclose).
For conferences, this implies:
- Baseline requirements for AI‑writing tools used in submissions.
- Safe harbors for disclosed AI use that passes integrity checks.
- Proportional responses when venue‑provided tools misbehave.
AI platform incidents (OpenAI payment leaks, mis‑indexed private chats, Meta code leaks) show organizations treating LLMs as an integrity and privacy risk surface.[5] The same confidentiality–integrity–availability lens applies to research claims.
CISO‑oriented LLM security frameworks map AI‑specific threats to ISO and NIST controls.[6] Conferences can map:
- Hallucinated evidence → violations of research ethics and reproducibility.
- Poisoned literature tools → track‑wide integrity risk.
- Unlogged AI assistance → audit gaps during investigations.[3][6]
💼 Tooling as attack surface
2026 security wrap‑ups highlight LangChain/LangGraph CVEs across tens of millions of downloads, making orchestration layers active attack surfaces.[7][9] If authors depend on tools built on these stacks, those tools fall inside the venue’s trust boundary and governance scope.
Harris et al. show frontier labs prioritizing speed and scale over mature governance.[8] Conferences that adopt this culture without counter‑balancing rules risk embedding similar failures in the archival record.
4. A Multi‑Layer Defense Framework for AI‑Heavy Research Submissions
Hiriyanna and Zhao’s framework for high‑stakes LLMs can be adapted to four layers for conferences: author tools, submission checks, review enhancements, and post‑acceptance audits.[3]
4.1 Author‑tool layer
Authoring environments should enforce:[2][3]
- Citation verification: resolve DOIs/links; flag unresolved or suspicious entries.
- Retrieval grounding: generate summaries only from attached PDFs or curated corpora.
- Structured experiment logging: templates that tie claims to configs, seeds, and scripts.
⚡ Design principle
Any tool that can fabricate a citation must at minimum mark it as unverified or block export until a human confirms it.[2]
4.2 Submission layer
Conferences can require structured AI‑usage disclosures:[6]
- Models, versions, and tools used.
- Sections affected (writing, code, figures, analysis).
- Validation methods (manual checks, secondary models, replication).
ISO/IEC 42001‑aligned organizations already track similar AI‑management data for audits; adapting it to submission forms is straightforward.[6]
4.3 Review layer
Automated gates should support, not replace, human review:[3][10]
- Citation resolvers: batch‑check references; flag non‑existent works or odd patterns.
- Metric anomaly detection: compare results to public leaderboards; highlight implausible gains.
- Replication‑on‑demand: for borderline or high‑impact work, trigger artifact evaluation or lightweight reruns, analogous to CI/CD gates.
📊 Parallel from CI/CD
DevSecOps guidance treats AI‑generated code as untrusted, enforced by SAST, SCA, and policy gates.[10] AI‑authored experiments and analyses deserve the same “distrust and verify” stance.
4.4 Post‑acceptance layer
Venues should institutionalize:[5][7]
- Random audits of accepted papers (citation verification, selective reruns).
- Corrigendum and retraction workflows modeled on security‑incident post‑mortems, with root‑cause analysis feeding tool and policy updates.
💡 Measure the defenders
Legal hallucination benchmarks and AI‑risk surveys emphasize evaluating mitigation, not just specifying it.[2][8] Conferences should track:[3]
- Detection rates for hallucinated references and artifacts.
- False‑positive rates and reviewer overhead.
- Added latency and operational costs per submission.
5. Implementation Roadmap: Before ICLR 2027
5.1 Authors: Distrust and Verify
DevSecOps reports recommend treating all AI‑generated code as “tainted” until independently validated.[10] Authors should adopt the same stance toward AI‑generated text, tables, and figures:[1][10]
- Never include AI‑generated citations without confirming they exist.
- Re‑run any experiment the model “helped design”; record actual outputs.
- Maintain a private provenance log of prompts, drafts, and edits for potential audits.
⚠️ Red flag list for your own drafts
- References missing from all major databases.
- Benchmarks you have never seen elsewhere.
- Perfectly smooth tables with no variance or failed runs.
If ICLR 2026 exposed anything, it is that generative AI can silently erode the evidentiary fabric of research. Treating AI outputs as untrusted until verified—and aligning tools, policies, and incentives around that principle—is essential if flagship venues want to remain credible in an AI‑saturated publication ecosystem.[1][2][3]
About CoreProse: Research-first AI content generation with verified citations. Zero hallucinations.
