DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

arXiv cs.CV / 4/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that visual-prompted object detection underperforms because visual prompts lack global discriminability, even though visual prompts can outperform text prompts for rare categories.
It introduces DETR-ViP, a robust detection transformer framework that learns class-distinguishable visual prompts beyond standard image-text contrastive learning.
DETR-ViP improves visual prompt representations using global prompt integration and visual-textual prompt relation distillation.
It further uses a selective fusion strategy to stabilize and strengthen detection results.
Experiments on COCO, LVIS, ODinW, and Roboflow100 show DETR-ViP achieves substantially higher performance than existing state-of-the-art methods, supported by ablation studies and analyses.

Abstract

Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.