Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

arXiv cs.AI / 4/22/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper treats scientific feasibility assessment as a diagnostic reasoning task where an LLM predicts whether a hypothesis is feasible or infeasible and explains its decision.
Under controlled “knowledge conditions,” the authors test LLM performance with hypothesis-only inputs, experiment descriptions, outcome evidence, and combinations of these.
Across multiple LLMs and two datasets, outcome evidence is generally more reliable than experiment descriptions for improving feasibility judgments.
The study finds that outcomes improve accuracy beyond using the model’s internal knowledge alone, while experiment text can be brittle and hurt performance when the provided context is incomplete.
By systematically removing parts of experiment/outcome context, the work characterizes robustness and identifies when experimental evidence helps versus when it introduces fragility.

Abstract

Scientific feasibility assessment asks whether a claim is consistent with established knowledge and whether experimental evidence could support or refute it. We frame feasibility assessment as a diagnostic reasoning task in which, given a hypothesis, a model predicts feasible or infeasible and justifies its decision. We evaluate large language models (LLMs) under controlled knowledge conditions (hypothesis-only, with experiments, with outcomes, or both) and probe robustness by progressively removing portions of the experimental and/or outcome context. Across multiple LLMs and two datasets, providing outcome evidence is generally more reliable than providing experiment descriptions. Outcomes tend to improve accuracy beyond what internal knowledge alone provides, whereas experimental text can be brittle and may degrade performance when the context is incomplete. These findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.