LLM-Enhanced Log Anomaly Detection: A Comprehensive Benchmark of Large Language Models for Automated System Diagnostics

arXiv cs.LG / 4/15/2026

💬 OpinionIdeas & Deep AnalysisTools & Practical UsageModels & Research

共有:

Key Points

The paper introduces a comprehensive benchmark comparing LLM-based and traditional methods for system log anomaly detection across four public datasets (HDFS, BGL, Thunderbird, Spirit).
It evaluates three method families: classical log parsers plus ML classifiers, fine-tuned transformer models (BERT/RoBERTa), and prompt-based LLM approaches (GPT-3.5, GPT-4, LLaMA-3) in zero-shot and few-shot settings.
Fine-tuned transformers deliver the best accuracy, reaching F1 scores of about 0.96–0.99, while prompt-based LLMs still perform strongly in zero-shot (F1 roughly 0.82–0.91) without labeled training data.
The study includes analysis of practical deployment considerations including cost-accuracy trade-offs, latency, and common failure modes across approaches.
The authors release code and configurations to support reproducibility and provide practitioner-oriented guidelines for selecting methods under constraints like label scarcity, latency, and budget.

Abstract

System log anomaly detection is critical for maintaining the reliability of large-scale software systems, yet traditional methods struggle with the heterogeneous and evolving nature of modern log data. Recent advances in Large Language Models (LLMs) offer promising new approaches to log understanding, but a systematic comparison of LLM-based methods against established techniques remains lacking. In this paper, we present a comprehensive benchmark study evaluating both LLM-based and traditional approaches for log anomaly detection across four widely-used public datasets: HDFS, BGL, Thunderbird, and Spirit. We evaluate three categories of methods: (1) classical log parsers (Drain, Spell, AEL) combined with machine learning classifiers, (2) fine-tuned transformer models (BERT, RoBERTa), and (3) prompt-based LLM approaches (GPT-3.5, GPT-4, LLaMA-3) in zero-shot and few-shot settings. Our experiments reveal that while fine-tuned transformers achieve the highest F1-scores (0.96-0.99), prompt-based LLMs demonstrate remarkablezero-shot capabilities (F1: 0.82-0.91) without requiring any labeled training data -- a significant advantage for real-world deployment where labeled anomalies are scarce. We further analyze the cost-accuracy trade-offs, latency characteristics, and failure modes of each approach. Our findings provide actionable guidelines for practitioners choosing log anomaly detection methods based on their specific constraints regarding accuracy, latency, cost, and label availability. All code and experimental configurations are publicly available to facilitate reproducibility.