Sparse Regression under Correlation and Weak Signals: A Reproducible Benchmark of Classical and Bayesian Methods

arXiv cs.LG / 5/5/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper benchmarks classical sparse regression methods against Bayesian sparse regression approaches under realistic hard regimes like correlated features, weak signals, and increasing dimensionality.
It compares six techniques (OLS, Ridge, Lasso, Elastic Net, Horseshoe, and Spike-and-Slab) across synthetic datasets spanning multiple covariance structures (ρ up to 0.9), SNR levels, and dimensionalities (p = 20, 50, 100), plus the Diabetes dataset for 2,600+ experiments.
Bayesian methods generally outperform in prediction error, with reported MSEs showing clear gains (e.g., 72 for Bayesian vs. 108–267 for some classical counterparts).
The Horseshoe prior achieves close-to-nominal uncertainty calibration, delivering about 94.8% coverage for its 95% intervals.
For variable selection, Lasso and Spike-and-Slab achieve similar F1 scores (~0.47), leading the authors to recommend Lasso as the practical default when full posterior uncertainty is unnecessary, and they provide reproducible code and data.

Abstract

Choosing between classical and Bayesian sparse regression methods involves a real trade-off: penalized estimators like Lasso run in milliseconds but give no uncertainty estimates,while Horseshoe and Spike-and-Slab priors produce full posteriors but need MCMC chains that take minutes per fit.Surprisingly few studies compare these two families head-to-head under the conditions that actually make sparse regression hard -- correlated features, weak signals, and growing dimensionality. We benchmark six methods (OLS, Ridge,Lasso, Elastic Net, Horseshoe, Spike-and-Slab) on synthetic data with three covariance structures (rho up to 0.9), four SNR levels, and p in {20, 50, 100}, plus the Diabetes dataset,totalling over 2,600 experiments. The results are clear on some points and nuanced on others. Bayesian methods win on prediction error (MSE 72 vs. 108-267), and the Horseshoe delivers near-nominal 95% coverage (94.8%). But Spike-and-Slab,despite narrower intervals, under-covers at 91.9% -- its continuous relaxation likely plays a role. For variable selection, Lasso and Spike-and-Slab tie at F1 ~ 0.47, making Lasso the practical default when posteriors are not needed. Code and data are available at https://github.com/xiao98/sparse-bayesian-regression-bench.