Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

arXiv cs.AI / 4/6/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper addresses Audio-Visual Navigation (AVN) where binaural audio cues are intermittently unreliable, especially when the agent encounters previously unheard sound categories.
It proposes RAVN, a framework that conditions cross-modal fusion on audio-derived reliability cues to dynamically balance audio and visual inputs in complex acoustics.
RAVN includes an Acoustic Geometry Reasoner (AGR) trained with geometric proxy supervision using a heteroscedastic Gaussian negative log-likelihood objective to learn observation-dependent dispersion as a reliability cue without requiring geometric labels at inference.
It further introduces Reliability-Aware Geometric Modulation (RAGM), which turns the learned reliability cue into a soft gate that modulates visual features to reduce cross-modal conflicts.
Experiments on the SoundSpaces benchmark using Replica and Matterport3D show consistent navigation performance gains, with improved robustness in the challenging “unheard sound” generalization setting.

Abstract

Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.