Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCE

arXiv cs.LG / 3/16/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper models embedding evolution under Langevin dynamics on a compact Riemannian manifold to study how the temperature parameter affects InfoNCE in contrastive learning.
It shows that under mild smoothness and energy-barrier assumptions, slow logarithmic inverse-temperature schedules guarantee convergence in probability to globally optimal representations, extending simulated annealing guarantees to this setting.
It warns that faster temperature schedules risk getting trapped in suboptimal minima, highlighting a trade-off between exploration and convergence.
The work establishes a principled link between contrastive learning and simulated annealing, offering guidance on how to tune temperature schedules in practice.

Abstract

The InfoNCE loss in contrastive learning depends critically on a temperature parameter, yet its dynamics under fixed versus annealed schedules remain poorly understood. We provide a theoretical analysis by modeling embedding evolution under Langevin dynamics on a compact Riemannian manifold. Under mild smoothness and energy-barrier assumptions, we show that classical simulated annealing guarantees extend to this setting: slow logarithmic inverse-temperature schedules ensure convergence in probability to a set of globally optimal representations, while faster schedules risk becoming trapped in suboptimal minima. Our results establish a link between contrastive learning and simulated annealing, providing a principled basis for understanding and tuning temperature schedules.