Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism

arXiv cs.CV / 4/1/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper proposes FlexMem, a training-free visual memory mechanism to improve long-form video understanding for multimodal large language models (MLLMs) beyond typical input length limits.
FlexMem treats visual KV caches as memory sources and uses a dual-pathway compression design to enable efficient memory transfer and writing as video context grows.
It investigates multiple memory reading strategies tailored to different video understanding tasks, including streaming-style scenarios.
Experiments on two video-MLLMs across five long-video and one streaming dataset show substantial gains versus existing efficient methods, including processing more than 1,000 frames on a single RTX 3090 GPU.
The approach can also strengthen base MLLMs, producing comparable or better benchmark performance than state-of-the-art proprietary models on some tasks (e.g., GPT-4o and Gemini-1.5 Pro).

Abstract

Long video understanding is a key challenge that plagues the advancement of \emph{Multimodal Large language Models} (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed \emph{Flexible Memory} (\textbf{FlexMem}). In principle, FlexMem aims to mimic human behavior of video watching, \emph{i.e.}, continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on \textbf{a single 3090 GPU}, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than \textbf{1k frames}, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, \emph{e.g.} , GPT-4o and Gemini-1.5 Pro.