A Complete Symmetry Classification of Shallow ReLU Networks

arXiv cs.LG / 4/16/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that for neural networks, parameter space cannot always be treated as equivalent to function space, motivating the study of symmetries among parameters that yield the same function.
It frames this via the “neuromanifold,” a quotient space identifying function-equivalent parameters and linking it to geometric properties that can affect optimization dynamics.
Prior symmetry classification approaches often required activation-function analyticity, which left out the important case of ReLU.
By leveraging ReLU’s non-differentiability, the authors provide a complete symmetry classification for shallow ReLU networks.

Abstract

Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as ``reverse engineering," or ``parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the \textit{neuromanifold}, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.