JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

arXiv cs.AI / 3/25/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

Key Points

  • The paper proposes a textual representation that converts fault trees stored as images into a format that LLMs can directly process for malfunction tracking and analysis.
  • It introduces JFTA-Bench, a multi-turn dialogue benchmark with 3,130 fault-related entries and an average of 40.75 dialogue turns per entry, focused on assisting malfunction localization in complex environments.
  • The authors train an end-to-end model to generate deliberately vague information to better mirror realistic user behavior.
  • To test robustness under human error, the benchmark includes long-range rollback and recovery procedures that simulate user mistakes and require error recovery.
  • The results report Gemini 2.5 Pro as achieving the best performance on the benchmark.

Abstract

In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains 3130 entries and 40.75 turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance.