Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descent

arXiv stat.ML / 4/1/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper presents two computationally cheap, resampling-based techniques for constructing confidence intervals and uncertainty estimates for solutions produced by stochastic gradient descent (SGD).
One method runs a small number of SGD instances in parallel using bootstrap resampling with replacement, while the other applies a related strategy in an online setting.
The approach reframes the methods as improved versions of established bootstrap procedures, targeting major reductions in resampling cost and avoiding complex mixing requirements found in some prior batching analyses.
The authors leverage a “cheap bootstrap” concept and refine a Berry–Esseen-type bound tailored to SGD to justify the statistical guarantees.

Abstract

Stochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained solutions from SGD has only been recently studied, yet it is important due to the growing need for uncertainty quantification. We investigate two computationally cheap resampling-based methods to construct confidence intervals for SGD solutions. One uses multiple, but few, SGDs in parallel via resampling with replacement from the data, and another operates this in an online fashion. Our methods can be regarded as enhancements of established bootstrap schemes to substantially reduce the computation effort in terms of resampling requirements, while bypassing the intricate mixing conditions in existing batching methods. We achieve these via a recent so-called cheap bootstrap idea and refinement of a Berry-Esseen-type bound for SGD.