Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum

arXiv cs.CL / 3/30/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that supervised fine-tuning (SFT)’s common negative log likelihood (NLL) objective can underperform because post-training violates NLL’s “train-from-scratch” optimality assumptions under noisy or long supervision.
Instead of proposing one loss, it systematically evaluates multiple probability-based training objectives and analyzes when they succeed or fail across different settings.
Experiments covering 8 model backbones, 27 benchmarks, and 7 domains show a key governing factor: the model-capability continuum, where prior-leaning objectives that downweight low-probability tokens outperform NLL for strong models.
For weaker models, NLL is found to be more effective, while intermediate-capability models show no single objective consistently winning.
A theoretical analysis explains how different objectives “trade places” across the continuum and provides guidance for selecting/adapting objectives based on model capability, with code released on GitHub.

Abstract

Supervised fine-tuning (SFT) is the standard approach for post-training large language models (LLMs), yet it often shows limited generalization. We trace this limitation to its default training objective: negative log likelihood (NLL). While NLL is classically optimal when training from scratch, post-training operates in a different paradigm and could violate its optimality assumptions, where models already encode task-relevant priors and supervision can be long and noisy. Rather than proposing a single universally superior replacement loss, we systematically study various probability-based objectives and characterize when and why different objectives succeed or fail under varying conditions. Through comprehensive experiments and extensive ablation studies across 8 model backbones, 27 benchmarks, and 7 domains, we uncover a critical dimension that governs objective behavior: the model-capability continuum. Near the model-strong end, prior-leaning objectives that downweight low-probability tokens (e.g.,

-p

-p^{10}

, thresholded variants) consistently outperform NLL; toward the model-weak end, NLL dominates; in between, no single objective prevails. Our theoretical analysis further elucidates how objectives trade places across the continuum, providing a principled foundation for adapting objectives to model capability. The code is provided at https://github.com/GaotangLi/Beyond-Log-Likelihood.