The Conjugate Domain Dichotomy: Exact Risk of M-Estimators under Infinite-Variance Noise in High Dimensions

arXiv stat.ML / 3/31/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper analyzes high-dimensional M-estimation in the proportional asymptotic regime (p/n → γ > 0) under infinite-variance noise with regularly varying tails (tail index α ∈ (1,2)).
It shows that the asymptotic risk behavior of a regularized M-estimator is determined by a geometric property of the loss function: whether the domain of its Fenchel conjugate is bounded or unbounded.
For bounded conjugate-domain losses (e.g., Huber, absolute-value, and quantile), the dual variable is effectively constrained, the relevant noise impact reduces to a finite first absolute moment, and the estimator attains bounded risk without external/transfer information.
For unbounded conjugate-domain losses such as squared loss, the dual variable grows with the noise, the risk depends on the diverging second moment, and bounded risk requires transfer regularization toward an external prior.
For the squared-loss case, the authors derive the exact asymptotic risk using the Convex Gaussian Minimax Theorem with noise-adapted regularization, leading to a trichotomy: non-transfer squared-loss risk diverges, Huber-style boundedness yields non-vanishing risk, and transfer-regularized methods reach a universal risk floor.

Abstract

This paper studies high-dimensional M-estimation in the proportional asymptotic regime (p/n -> gamma > 0) when the noise distribution has infinite variance. For noise with regularly-varying tails of index alpha in (1,2), we establish that the asymptotic behavior of a regularized M-estimator is governed by a single geometric property of the loss function: the boundedness of the domain of its Fenchel conjugate. When this conjugate domain is bounded -- as is the case for the Huber, absolute-value, and quantile loss functions -- the dual variable in the min-max formulation of the estimator is confined, the effective noise reduces to the finite first absolute moment of the noise distribution, and the estimator achieves bounded risk without recourse to external information. When the conjugate domain is unbounded -- as for the squared loss -- the dual variable scales with the noise, the effective noise involves the diverging second moment, and bounded risk can be achieved only through transfer regularization toward an external prior. For the squared-loss class specifically, we derive the exact asymptotic risk via the Convex Gaussian Minimax Theorem under a noise-adapted regularization scaling. The resulting risk converges to a universal floor that is independent of the regularizer, yielding a loss-risk trichotomy: squared-loss estimators without transfer diverge; Huber-loss estimators achieve bounded but non-vanishing risk; transfer-regularized estimators attain the floor.