The Bernstein-von Mises theorem for Bayesian one-pass online learning

arXiv stat.ML / 5/1/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies Bayesian online learning in the challenging one-pass setting, where standard theory breaks down because it usually requires mini-batch sizes to grow without bound.
It introduces a new one-pass Bayesian online learning algorithm that uses a warm-start phase to stabilize sequential posterior updates.
The authors prove that the sequentially updated posterior achieves the optimal convergence rate under their method.
They further derive an online version of the Bernstein–von Mises theorem, providing reliable uncertainty quantification without needing diverging mini-batch sample sizes.
Experiments on generalized linear models indicate the method’s performance is comparable to batch estimation and better than existing online approaches.

Abstract

Bayesian online learning provides a coherent framework for sequential inference. However, its theoretical understanding remains limited, particularly in the one-pass setting. Existing theoretical guarantees typically require the mini-batch sample size to diverge, a condition that fails in the one-pass regime. In this paper, we propose a new Bayesian online learning algorithm tailored to the one-pass setting, which incorporates a warm-start phase to ensure stable sequential updates. For this algorithm, we show that the sequentially updated posterior attains the optimal convergence rate. Building on this, we establish an online analogue of the Bernstein-von Mises theorem, which guarantees valid uncertainty quantification without diverging mini-batch sample sizes. Our analysis is based on a novel theoretical framework that differs fundamentally from existing approaches in the online learning literature. Numerical experiments on generalized linear models show that the proposed method matches the performance of the batch estimator while outperforming existing online procedures.