Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms

arXiv cs.CV / 4/22/2026

📰 NewsDeveloper Stack & InfrastructureModels & Research

共有:

Key Points

The paper argues that Vision Transformer (ViT) models underperform in computer-aided breast cancer diagnosis because mammograms require attending to small, high-resolution abnormalities and because the task is inherently fine-grained with subtle class differences.
It identifies two key failure modes: token overload that hampers softmax attention localization, and limited discriminative power from standard cross-entropy training under low inter-class and high intra-class variability.
To address these issues, the authors propose a three-part framework: RoI-based token reduction using an object detector, hard-negative contrastive learning between selected RoIs, and a localization-leaning DINOv2-pretrained ViT instead of global CLIP-style features.
Experiments on public mammography datasets show improved performance over existing baselines, suggesting the approach could be useful for large-scale breast cancer screening.
The authors release code to support reproducibility, enabling others to evaluate and extend the method.

Abstract

Vision Transformers

(\texttt{ViT})

have become the architecture of choice for many computer vision tasks, yet their performance in computer-aided diagnostics remains limited. Focusing on breast cancer detection from mammograms, we identify two main causes for this shortfall. First, medical images are high-resolution with small abnormalities, leading to an excessive number of tokens and making it difficult for the softmax-based attention to localize and attend to relevant regions. Second, medical image classification is inherently fine-grained, with low inter-class and high intra-class variability, where standard cross-entropy training is insufficient. To overcome these challenges, we propose a framework with three key components: (1) Region of interest

(\texttt{RoI})

based token reduction using an object detection model to guide attention; (2) contrastive learning between selected

\texttt{RoI}

to enhance fine-grained discrimination through hard-negative based training; and (3) a

\texttt{DINOv2}

pretrained

\texttt{ViT}

that captures localization-aware, fine-grained features instead of global

\texttt{CLIP}

representations. Experiments on public mammography datasets demonstrate that our method achieves superior performance over existing baselines, establishing its effectiveness and potential clinical utility for large-scale breast cancer screening. Our code is available for reproducibility here: https://aih-iitd.github.io/publications/attend-what-matters