When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

arXiv cs.CL / 4/24/2026

📰 NewsIdeas & Deep AnalysisModels & Research

Key Points

  • The paper argues that LLM model distillation can cause “behavioral homogenization” in tool-using agents, making many systems share similar reasoning steps and failure modes.
  • It proposes two new metrics to separate mandatory success behaviors from optional preference-driven patterns: Response Pattern Similarity (RPS) for text and Action Graph Similarity (AGS) for tool-use represented as directed graphs.
  • Using evaluations of 18 models from 8 providers on τ-Bench and τ²-Bench against Claude Sonnet 4.5, the authors find higher within-family similarity on AGS than cross-family, suggesting teacher-induced convergence.
  • A controlled distillation experiment supports that AGS can distinguish teacher-specific convergence from general performance improvement, and RPS/AGS are shown to capture different behavioral aspects (Pearson r = 0.491).
  • The work provides accompanying code via GitHub to enable further analysis of behavioral convergence in the agent ecosystem.

Abstract

Model distillation is a primary driver behind the rapid progress of LLM agents, yet it often leads to behavioral homogenization. Many emerging agents share nearly identical reasoning steps and failure modes, suggesting they may be distilled echoes of a few dominant teachers. Existing metrics, however, fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences. We propose two complementary metrics to isolate non-mandatory behavioral patterns: \textbf{Response Pattern Similarity (RPS)} for verbal alignment and \textbf{Action Graph Similarity (AGS)} for tool-use habits modeled as directed graphs. Evaluating 18 models from 8 providers on \tau-Bench and \tau^2-Bench against Claude Sonnet 4.5 (thinking), we find that within-family model pairs score 5.9 pp higher in AGS than cross-family pairs, and that Kimi-K2 (thinking) reaches 82.6\% S_{\text{node}} and 94.7\% S_{\text{dep}}, exceeding Anthropic's own Opus 4.1. A controlled distillation experiment further confirms that AGS distinguishes teacher-specific convergence from general improvement. RPS and AGS capture distinct behavioral dimensions (Pearson r = 0.491), providing complementary diagnostic signals for behavioral convergence in the agent ecosystem. Our code is available at https://github.com/Syuchin/AgentEcho.