PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization

arXiv cs.CV / 4/8/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper addresses a common multimodal learning issue where multimodal models can underperform compared to their unimodal counterparts due to insufficient optimization during training.
It challenges the prevailing assumption that “balanced” modality learning is optimal, arguing instead that performance should be dominated by the modality with superior unimodal performance.
The proposed PDMP method identifies the performance-dominant modality using rankings from independently trained unimodal models, without needing knowledge of multimodal model structures or fusion approaches.
PDMP then applies asymmetric gradient modulation coefficients so the dominant modality drives optimization, with the claim that under-optimization stems from insufficient learning of this modality.
Experiments across multiple datasets reportedly validate that PDMP improves multimodal performance.

Abstract

Multimodal learning has attracted increasing attention due to its practicality. However, it often suffers from insufficient optimization, where the multimodal model underperforms even compared to its unimodal counterparts. Existing methods attribute this problem to the imbalanced learning between modalities and solve it by gradient modulation. This paper argues that balanced learning is not the optimal setting for multimodal learning. On the contrary, imbalanced learning driven by the performance-dominant modality that has superior unimodal performance can contribute to better multimodal performance. And the under-optimization problem is caused by insufficient learning of the performance-dominant modality. To this end, we propose the Performance-Dominant Modality Prioritization (PDMP) strategy to assist multimodal learning. Specifically, PDMP firstly mines the performance-dominant modality via the performance ranking of the independently trained unimodal model. Then PDMP introduces asymmetric coefficients to modulate the gradients of each modality, enabling the performance-dominant modality to dominate the optimization. Since PDMP only relies on the unimodal performance ranking, it is independent of the structures and fusion methods of the multimodal model and has great potential for practical scenarios. Finally, extensive experiments on various datasets validate the superiority of PDMP.