HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

arXiv cs.AI / 3/12/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

HEAL is an RL-free framework for distilling reasoning from large reasoning models into smaller models, addressing rejection sampling limitations and the teacher ceiling.
It combines three modules: Guided Entropy-Assisted Repair (GEAR), Perplexity-Uncertainty Ratio Estimator (PURE), and Progressive Answer-guided Curriculum Evolution (PACE) to detect critical reasoning breakpoints, filter genuine breakthroughs, and guide curriculum progression.
The framework draws on the Zone of Proximal Development to inject hindsight hints and repair broken reasoning trajectories during training.
Extensive experiments on multiple benchmarks show that HEAL significantly outperforms traditional supervised fine-tuning distillation and other baselines.
This work presents a new approach in model distillation and demonstrates notable improvements over standard methods.

Abstract

Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitation of rejection sampling. Standard methods treat the teacher as a static filter, discarding complex "corner-case" problems where the teacher fails to explore valid solutions independently, thereby creating an artificial "Teacher Ceiling" for the student. In this work, we propose Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap. Drawing on the educational theory of the Zone of Proximal Development(ZPD), HEAL synergizes three core modules: (1) Guided Entropy-Assisted Repair (GEAR), an active intervention mechanism that detects critical reasoning breakpoints via entropy dynamics and injects targeted hindsight hints to repair broken trajectories; (2) Perplexity-Uncertainty Ratio Estimator (PURE), a rigorous filtering protocol that decouples genuine cognitive breakthroughs from spurious shortcuts; and (3) Progressive Answer-guided Curriculum Evolution (PACE), a three-stage distillation strategy that organizes training from foundational alignment to frontier breakthrough. Extensive experiments on multiple benchmarks demonstrate that HEAL significantly outperforms traditional SFT distillation and other baselines.