MOSA: Motion-Guided Semantic Alignment for Dynamic Scene Graph Generation

arXiv cs.CV / 4/22/2026

📰 NewsModels & Research

共有:

Key Points

The paper introduces MOSA, a motion-guided semantic alignment approach for Dynamic Scene Graph Generation that targets shortcomings in fine-grained and tail relationship modeling.
It encodes object-pair motion attributes (e.g., distance, velocity, motion persistence, directional consistency) using a Motion Feature Extractor and fuses them with spatial relationship features via a Motion-guided Interaction Module.
To improve semantic discrimination, MOSA uses a cross-modal Action Semantic Matching mechanism that aligns visual relationship features with text embeddings of relationship categories.
It adds a category-weighted loss to emphasize learning of infrequent (“tail”) relationships, and reports the best performance on the Action Genome dataset.

Abstract

Dynamic Scene Graph Generation (DSGG) aims to structurally model objects and their dynamic interactions in video sequences for high-level semantic understanding. However, existing methods struggle with fine-grained relationship modeling, semantic representation utilization, and the ability to model tail relationships. To address these issues, this paper proposes a motion-guided semantic alignment method for DSGG (MoSA). First, a Motion Feature Extractor (MFE) encodes object-pair motion attributes such as distance, velocity, motion persistence, and directional consistency. Then, these motion attributes are fused with spatial relationship features through the Motion-guided Interaction Module (MIM) to generate motion-aware relationship representations. To further enhance semantic discrimination capabilities, the cross-modal Action Semantic Matching (ASM) mechanism aligns visual relationship features with text embeddings of relationship categories. Finally, a category-weighted loss strategy is introduced to emphasize learning of tail relationships. Extensive and rigorous testing shows that MoSA performs optimally on the Action Genome dataset.