A Compression Perspective on Simplicity Bias

arXiv cs.AI / 3/30/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper analyzes the “simplicity bias” in deep neural networks—its tendency to prefer simple functions—by reframing supervised learning as Minimum Description Length (MDL)-style two-part lossless compression.
It proposes a formal trade-off between hypothesis (model) complexity cost and data encoding/predictive cost, explaining how this trade-off drives neural networks’ feature selection.
The framework predicts qualitative transitions in learned features as training data increases, moving from simple spurious shortcuts toward more complex “real” features only when the data-encoding savings outweigh added model complexity.
It identifies different data regimes: more data can improve robustness by eliminating trivial shortcuts, while in other regimes limiting data can act like complexity-based regularization to avoid unreliable cues.
The authors validate the theory on a semi-synthetic benchmark, finding that neural feature selection follows the solution trajectory of optimal two-part compressors.

Abstract

Deep neural networks exhibit a simplicity bias, a well-documented tendency to favor simple functions over complex ones. In this work, we cast new light on this phenomenon through the lens of the Minimum Description Length principle, formalizing supervised learning as a problem of optimal two-part lossless compression. Our theory explains how simplicity bias governs feature selection in neural networks through a fundamental trade-off between model complexity (the cost of describing the hypothesis) and predictive power (the cost of describing the data). Our framework predicts that as the amount of available training data increases, learners transition through qualitatively different features -- from simple spurious shortcuts to complex features -- only when the reduction in data encoding cost justifies the increased model complexity. Consequently, we identify distinct data regimes where increasing data promotes robustness by ruling out trivial shortcuts, and conversely, regimes where limiting data can act as a form of complexity-based regularization, preventing the learning of unreliable complex environmental cues. We validate our theory on a semi-synthetic benchmark showing that the feature selection of neural networks follows the same trajectory of solutions as optimal two-part compressors.