MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment

arXiv cs.LG / 4/23/2026

📰 NewsIdeas & Deep AnalysisTools & Practical UsageModels & Research

共有:

Key Points

The paper addresses LLM alignment as a multi-objective optimization problem involving potentially conflicting goals like helpfulness, truthfulness, and harmlessness.
It argues that using a fixed scalarization in many alignment pipelines can create procedural unfairness by under-weighting harder or minority objectives.
It proposes MGDA-Decoupled, a geometry-aware multi-objective algorithm that computes a shared descent direction while accounting for each objective’s convergence dynamics.
Unlike prior approaches that use reinforcement learning or explicit reward models, MGDA-Decoupled is designed to work entirely within the lightweight DPO (Direct Preference Optimization) framework.
Experiments on the UltraFeedback dataset indicate that geometry-aware methods, especially MGDA-Decoupled, achieve the highest win rates against golden responses overall and across individual objectives.

Abstract

Aligning large language models (LLMs) to desirable human values requires balancing multiple, potentially conflicting objectives such as helpfulness, truthfulness, and harmlessness, which presents a multi-objective optimisation challenge. Most alignment pipelines rely on a fixed scalarisation of these objectives, which can introduce procedural unfairness by systematically under-weighting harder-to-optimise or minority objectives. To promote more equitable trade-offs, we introduce MGDA-Decoupled, a geometry-based multi-objective optimisation algorithm that finds a shared descent direction while explicitly accounting for each objective's convergence dynamics. In contrast to prior methods that depend on reinforcement learning (e.g., GAPO) or explicit reward models (e.g., MODPO), our approach operates entirely within the lightweight Direct Preference Optimisation (DPO) paradigm. Experiments on the UltraFeedback dataset show that geometry-aware methods -- and MGDA-Decoupled in particular -- achieve the highest win rates against golden responses, both overall and per objective.