Early-stopped aggregation: Adaptive inference with computational efficiency

arXiv stat.ML / 4/17/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that adaptive statistical inference and model aggregation often require computing estimators for many unnecessarily complex models, causing major computational inefficiency when the true data-generating process is simple.
It introduces “early-stopped aggregation (ESA),” which aggregates results only from a small subset of simpler candidate models selected via an early-stopping criterion, reducing wasted computation.
The authors prove ESA’s adaptive optimality properties in variational Bayes under mild conditions, and they extend these results to variational empirical Bayes with data-dependent prior hyperparameters.
They also develop ESA theory for frequentist aggregation, including penalized estimation and sample-splitting variants, and show a unifying perspective where both early-stopped Bayes and penalized frequentist aggregation rely on a shared “energy” functional (data fit plus complexity control).
The work reports applications and numerical studies indicating that ESA can deliver both computational efficiency and strong estimation performance across multiple learning settings.

Abstract

When considering a model selection or, more generally, an aggregation approach for adaptive statistical inference, it is often necessary to compute estimators over a wide range of model complexities including unnecessarily large models even when the true data-generating process is relatively simple, due to the lack of prior knowledge. This requirement can lead to substantial computational inefficiency. In this work, we propose a novel framework for efficient model aggregation called the early-stopped aggregation (ESA): instead of computing and aggregating estimators for all candidate models, we compute only a small number of simpler ones using an early-stopping criterion and aggregate only these for final inference. Our framework is versatile and applies to both Bayesian model selection, in particular, within the variational Bayes framework, and frequentist estimation, including a general penalized estimation setting. We investigate adaptive optimal property of the ESA approach across three learning paradigms. We first show that ESA achieves optimal adaptive contraction rates in the variational Bayes setting under mild conditions. We extend this result to variational empirical Bayes, where prior hyperparameters are chosen in a data-dependent manner. In addition, we apply the ESA approach to frequentist aggregation including both penalization-based and sample-splitting implementations, and establish corresponding theory. As we demonstrate, there is a clear unification between early-stopped Bayes and frequentist penalized aggregation, with a common "energy" functional comprising a data-fitting term and a complexity-control term that drives both procedures. We further present several applications and numerical studies that highlight the efficiency and strong performance of the proposed approach.