Class Unlearning via Depth-Aware Removal of Forget-Specific Directions

arXiv cs.CV / 4/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper highlights that existing class-unlearning methods may not achieve true forgetting, since forget-class information can remain in internal representations or be masked primarily by changes to the classifier head.
It reports that prior approaches often show weak/negative selectivity, preserve forget-class structure in deep representations, or depend heavily on final-layer bias shifts.
The authors propose DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form “weight-surgery” technique that removes forget-specific directions via projection rather than gradient-based retraining.
DAMP computes class prototypes per stage, extracts forget directions as residuals relative to retain prototypes, and applies depth-aware scaling to make smaller edits in early layers and larger edits in deeper layers.
Experiments on MNIST, CIFAR-10/100, and Tiny ImageNet (across CNNs and transformers) suggest DAMP aligns more closely with full retraining, improving selective forgetting while better preserving performance on retain classes.

Abstract

Machine unlearning aims to remove targeted knowledge from a trained model without the cost of retraining from scratch. In class unlearning, however, reducing accuracy on forget classes does not necessarily imply true forgetting: forgotten information can remain encoded in internal representations, and apparent forgetting may arise from classifier-head suppression rather than representational removal. We show that existing class-unlearning methods often exhibit weak or negative selectivity, preserve forget-class structure in deep representations, or rely heavily on final-layer bias shifts. We then introduce DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form weight-surgery method that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, DAMP computes class prototypes in the input space of the next learnable operator, extracts forget directions as residuals relative to retain-class prototypes, and applies a projection-based update to reduce downstream sensitivity to those directions. To preserve utility, DAMP uses a parameter-free depth-aware scaling rule derived from probe separability, applying smaller edits in early layers and larger edits in deeper layers. The method naturally extends to multi-class forgetting through low-rank subspace removal. Across MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, and across convolutional and transformer architectures, DAMP more closely resembles the retraining gold standard than some of the prior methods, improving selective forgetting while better preserving retain-class performance and reducing residual forget-class structure in deep layers.