From Scene to Object: Text-Guided Dual-Gaze Prediction

arXiv cs.CV / 4/23/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that existing driver attention datasets lack object-level gaze annotations, which makes text-grounded cognitive modeling difficult and can cause text-vision decoupling and visual-bias hallucinations in VLMs.
To address this, it introduces G-W3DA, an object-level driver attention dataset built by combining a multimodal large language model with SAM3 to convert scene-level heatmaps into object-level masks while reducing annotation hallucinations through rigorous cross-validation.
It also proposes DualGaze-VLM, a dual-branch architecture that uses semantic-query hidden states and a Condition-Aware SE-Gate to dynamically modulate visual features for intent-driven, spatially anchored predictions.
Experiments on the W3DA benchmark show that DualGaze-VLM improves spatial alignment over prior SOTA, achieving up to a 17.8% gain in Similarity (SIM) in safety-critical scenarios.
A “visual Turing test” indicates that the generated attention heatmaps are judged authentic by 88.22% of human evaluators, suggesting the model can produce reasonable cognitive priors.

Abstract

Interpretable driver attention prediction is crucial for human-like autonomous driving. However, existing datasets provide only scene-level global gaze rather than fine-grained object-level annotations, inherently failing to support text-grounded cognitive modeling. Consequently, while Vision-Language Models (VLMs) hold great potential for semantic reasoning, this critical data limitations leads to severe text-vision decoupling and visual-bias hallucinations. To break this bottleneck and achieve precise object-level attention prediction, this paper proposes a novel dual-branch gaze prediction framework, establishing a complete paradigm from data construction to model architecture. First, we construct G-W3DA, a object-level driver attention dataset. By integrating a multimodal large language model with the Segment Anything Model 3 (SAM3), we decouple macroscopic heatmaps into object-level masks under rigorous cross-validation, fundamentally eliminating annotation hallucinations. Building upon this high-quality data foundation, we propose the DualGaze-VLM architecture. This architecture extracts the hidden states of semantic queries and dynamically modulates visual features via a Condition-Aware SE-Gate, achieving intent-driven precise spatial anchoring. Extensive experiments on the W3DA benchmark demonstrate that DualGaze-VLM consistently surpasses existing state-of-the-art (SOTA) models in spatial alignment metrics, notably achieving up to a 17.8% improvement in Similarity (SIM) under safety-critical scenarios. Furthermore, a visual Turing test reveals that the attention heatmaps generated by DualGaze-VLM are perceived as authentic by 88.22% of human evaluators, proving its capability to generate rational cognitive priors.