The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

arXiv cs.AI / 3/27/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that evaluating AI assistants in safety engineering is fundamentally difficult because “safety competence” is multidimensional, context-dependent, and subject to incompleteness and expert disagreement.
It proposes a formal five-dimensional competence framework covering domain knowledge, standards expertise, operational experience, contextual understanding, and judgment.
The authors introduce the “competence shadow,” defined as systematic narrowing of human reasoning caused by what AI analysis prevents experts from considering.
They model four human–AI collaboration structures and derive closed-form performance bounds showing the competence shadow can compound multiplicatively, creating degradation larger than simple additive expectations.
The work concludes that AI assistance quality in safety engineering is primarily a workflow/design problem (workflow qualification and shadow-resistant collaboration) rather than a tool/software procurement choice.

Abstract

As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.