Structural Graph Probing of Vision-Language Models

arXiv cs.CV / 3/31/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper analyzes vision-language models by treating each layer as a within-layer correlation graph built from neuron co-activations, enabling a “neural topology” view of computation.
It finds that correlation topology contains recoverable behavioral signal and varies systematically across modalities and network depth.
The study shows cross-modal structure becomes more consolidated with depth around a compact set of recurrent hub neurons.
Targeted perturbation of these hub neurons leads to substantial changes in model outputs, suggesting the identified components are causally influential.
The authors position neural topology as a useful interpretability intermediate—more informative than local attribution while being more tractable than full circuit-level recovery—and release code publicly.

Abstract

Vision-language models (VLMs) achieve strong multimodal performance, yet how computation is organized across populations of neurons remains poorly understood. In this work, we study VLMs through the lens of neural topology, representing each layer as a within-layer correlation graph derived from neuron-neuron co-activations. This view allows us to ask whether population-level structure is behaviorally meaningful, how it changes across modalities and depth, and whether it identifies causally influential internal components under intervention. We show that correlation topology carries recoverable behavioral signal; moreover, cross-modal structure progressively consolidates with depth around a compact set of recurrent hub neurons, whose targeted perturbation substantially alters model output. Neural topology thus emerges as a meaningful intermediate scale for VLM interpretability: richer than local attribution, more tractable than full circuit recovery, and empirically tied to multimodal behavior. Code is publicly available at https://github.com/he-h/vlm-graph-probing.