Implicit Bias in Deep Linear Discriminant Analysis

arXiv stat.ML / 4/13/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper analyzes implicit bias/regularization effects of discriminative metric-learning objectives, focusing on Deep LDA, a scale-invariant method aimed at reducing within-class variance and increasing between-class separation.
It provides a theoretical study of gradient flow in an L-layer diagonal linear network, showing how balanced initialization changes the effective form of updates during optimization.
The authors prove that, in this setting, standard additive gradient updates are transformed into multiplicative weight updates.
The result implies an “automatic conservation” of the (2/L) quasi-norm, linking optimization dynamics to implicit regularization behavior.
Overall, the work argues that the optimization geometry of Deep LDA remains underexplored and supplies an initial theoretical step toward characterizing it.

Abstract

While the Implicit Bias(or Implicit Regularization) of standard loss functions has been studied, the optimization geometry induced by discriminative metric-learning objectives remains largely unexplored.To the best of our knowledge, this paper presents an initial theoretical analysis of the implicit regularization induced by the Deep LDA,a scale invariant objective designed to minimize intraclass variance and maximize interclass distance. By analyzing the gradient flow of the loss on a L-layer diagonal linear network, we prove that under balanced initialization, the network architecture transforms standard additive gradient updates into multiplicative weight updates, which demonstrates an automatic conservation of the (2/L) quasi-norm.