Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

arXiv cs.AI / 4/1/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that current multi-agent red-team benchmarks cannot measure AI agents’ ability to support more autonomous SOCs because real SOC work is primarily blue-team oriented.
It claims no systematic benchmark has been proposed for coordinated multi-task blue-team evaluation of multi-agent AI, motivating a new benchmarking effort.
The authors propose design principles for constructing a benchmark called SOC-bench, focused on blue team capabilities rather than single-task assessments.
SOC-bench is presented as a family of five tasks centered on large-scale ransomware incident response, aiming to evaluate coordinated blue-team multi-agent performance.
The work provides a conceptual benchmark design rather than reporting a completed evaluation system, positioning it as a framework for future benchmark implementation and study.

Abstract

As Large Language Models (LLMs) and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to achieve more autonomous SOCs (security operation centers) and reduce manual effort. In particular, the AI and cybersecurity communities have recently developed several benchmarks for evaluating the red team capabilities of multi-agent AI systems. However, because the operations in SOCs are dominated by blue team operations, the capabilities of AI systems & agents to achieve more autonomous SOCs cannot be evaluated without a benchmark focused on blue team operations. To our best knowledge, no systematic benchmark for evaluating coordinated multi-task blue team AI has been proposed in the literature. Existing blue team benchmarks focus on a particular task. The goal of this work is to develop a set of design principles for the construction of a benchmark, which is denoted as SOC-bench, to evaluate the blue team capabilities of AI. Following these design principles, we have developed a conceptual design of SOC-bench, which consists of a family of five blue team tasks in the context of large-scale ransomware attack incident response.