MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

arXiv cs.CL / 3/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

MMKU-Bench is a comprehensive evaluation benchmark for multimodal knowledge updating, featuring over 25k knowledge instances and more than 49k images across updated and unknown knowledge scenarios.
The study shows that supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) tend to cause catastrophic forgetting when updating knowledge, while knowledge editing (KE) better preserves general capabilities but struggles with continual updating.
The benchmark enables cross-modal consistency assessment and systematic analysis across modalities, advancing evaluation methodology in multimodal knowledge updating.
The authors compare representative approaches (SFT, RLHF, KE) on MMKU-Bench, providing empirical insights into the strengths and limitations of each method.
Overall, MMKU-Bench offers a reliable platform for evaluating and guiding progress in multimodal knowledge updating.

Abstract

As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field.