Audio Spatially-Guided Fusion for Audio-Visual Navigation

arXiv cs.AI / 4/6/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies audio-visual navigation in 3D environments, focusing on robust target localization and path planning when environments and sound sources change beyond training data.
It proposes an “Audio Spatially-Guided Fusion” approach that uses an audio intensity attention mechanism to encode target-related spatial audio features.
The method introduces an Audio Spatial State Guided Fusion (ASGF) module to dynamically align and adaptively fuse multimodal (audio and visual) features.
Experiments on Replica and Matterport3D show improved generalization on “unheard” tasks, particularly under previously unseen sound-source distributions, suggesting reduced sensitivity to perceptual uncertainty and noise.

Abstract

Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.