A Regression Framework for Understanding Prompt Component Impact on LLM Performance

arXiv cs.LG / 3/31/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper proposes a statistical regression framework to quantify how specific prompt components affect LLM performance by linking prompt segments to evaluation outcomes.
It adapts prior XAI-style methods for LLM inspection, using regression on prompt portions to explain performance variation.
When applied to arithmetic tasks, the regression models explain 72% of performance variation for Mistral-7B and 77% for GPT-OSS-20B.
The authors find that misinformation in the form of incorrect example query-answer pairs harms both models’ ability to solve the arithmetic query, while positive examples show no significant variability in impact.
They report that positive and negative instructions can have contradictory effects on performance, highlighting the need for granular prompt auditing in critical deployments.

Abstract

As large language models (LLMs) continue to improve and see further integration into software systems, so does the need to understand the conditions in which they will perform. We contribute a statistical framework for understanding the impact of specific prompt features on LLM performance. The approach extends previous explainable artificial intelligence (XAI) methods specifically to inspect LLMs by fitting regression models relating portions of the prompt to LLM evaluation. We apply our method to compare how two open-source models, Mistral-7B and GPT-OSS-20B, leverage the prompt to perform a simple arithmetic problem. Regression models of individual prompt portions explain 72% and 77% of variation in model performances, respectively. We find misinformation in the form of incorrect example query-answer pairs impedes both models from solving the arithmetic query, though positive examples do not find significant variability in the impact of positive and negative instructions - these prompts have contradictory effects on model performance. The framework serves as a tool for decision makers in critical scenarios to gain granular insight into how the prompt influences an LLM to solve a task.