Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry

arXiv cs.LG / 5/1/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper investigates how dynamic adversarial fine-tuning (R2D2-style) reshapes the internal “refusal geometry” of a safety-aligned 7B language model during training.
Using a measurement-driven protocol that combines HarmBench, StrongREJECT, and XSTest with a five-anchor refusal-geometry suite and causal interventions, the authors track changes in jailbreak/refusal behavior over training steps.
Results show R2D2 can drive HarmBench attack success to near-zero at early-to-mid training (0.000 at steps 50 and 100), then partially increases later (0.035 at step 250 and 0.250 at step 500), while standard SFT stays much less robust (ASR ~0.505–0.588).
On XSTest, R2D2 maintains strong early “any-refusal” (1.000) but the metric declines substantially over time (0.664 and 0.228), indicating evolving refusal characteristics rather than a static defense.
The authors find that refusal “carriers” relocate from later-layer to earlier-layer representations during training while effective control rank stays roughly constant (~1.23–1.27), supporting a “reorganization” mechanism over a “drift-only” explanation.

Abstract

Safety-aligned language models must refuse harmful requests without collapsing into broad over-refusal, but the training-time mechanisms behind this tradeoff remain unclear. Prior work characterizes refusal directions and jailbreak robustness, yet does not explain how dynamic adversarial fine-tuning changes refusal carriers across training. We present a measurement-driven mechanism study, not a new defense, on one 7B backbone under supervised fine-tuning (SFT) and R2D2-style dynamic adversarial fine-tuning. Our protocol aligns fixed-source HarmBench, StrongREJECT, and XSTest with a five-anchor refusal-geometry suite and causal interventions. R2D2 drives fixed-source HarmBench ASR to 0.000 at steps 50 and 100, then partially reopens to 0.035 at step 250 and 0.250 at step 500; SFT remains less robust, with ASR between 0.505 and 0.588 at the same anchors. On XSTest, R2D2 any-refusal is 1.000 early, then falls to 0.664 and 0.228. Geometrically, R2D2 preserves a late-layer admissible carrier through step 100 before relocating to an early-layer carrier, while effective rank remains near 1.23--1.27. Causal interventions indicate low-dimensional but utility-coupled control. These results support a reorganization account rather than a drift-only account, with evidence limited to one backbone and fixed-source attacks.