GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining

arXiv cs.AI / 3/27/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

GoldiCLIP is a vision-language pretraining framework designed to balance different supervision signals (“Goldilocks” approach), targeting weaknesses in contrastive pretraining beyond simply using larger datasets.
It combines three components: text-conditioned self-distillation to align multiple feature types, an encoder–decoder architecture with a VQA objective to improve generalization beyond caption-like queries, and an uncertainty-based loss-weighting mechanism to balance heterogeneous training losses.
The method is trained on only 30M images (about 300× less data than leading approaches) yet achieves state-of-the-art results among data-efficient methods, improving MSCOCO retrieval by 2.2 points and question-based retrieval by 5.9 points over the best comparable baseline.
GoldiCLIP remains competitively close to billion-scale vision-language models, suggesting that better supervision design and loss balancing can offset large-scale data requirements.
The work is presented as an arXiv research announcement with a project page for details and reproducibility materials.

Abstract

Until recently, the success of large-scale vision-language models (VLMs) has primarily relied on billion-sample datasets, posing a significant barrier to progress. Latest works have begun to close this gap by improving supervision quality, but each addresses only a subset of the weaknesses in contrastive pretraining. We present GoldiCLIP, a framework built on a Goldilocks principle of finding the right balance of supervision signals. Our multifaceted training framework synergistically combines three key innovations: (1) a text-conditioned self-distillation method to align both text-agnostic and text-conditioned features; (2) an encoder integrated decoder with Visual Question Answering (VQA) objective that enables the encoder to generalize beyond the caption-like queries; and (3) an uncertainty-based weighting mechanism that automatically balances all heterogeneous losses. Trained on just 30 million images, 300x less data than leading methods, GoldiCLIP achieves state-of-the-art among data-efficient approaches, improving over the best comparable baseline by 2.2 points on MSCOCO retrieval, 2.0 on fine-grained retrieval, and 5.9 on question-based retrieval, while remaining competitive with billion-scale models. Project page: https://petsi.uk/goldiclip.