Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

arXiv cs.LG / 4/27/2026

📰 NewsDeveloper Stack & InfrastructureTools & Practical UsageModels & Research

共有:

Key Points

The paper addresses the high cost of fitting scaling laws, reframing it as a budget-aware sequential experimental design problem rather than a simple preprocessing step.
It proposes an uncertainty-aware strategy that selects which pilot experiments to run (from a heterogeneous-cost pool) to maximize extrapolation accuracy in a costly target training region.
Experiments on multiple scaling-law benchmark tasks show the method consistently beats classical design-based baselines.
The approach often achieves near full-data fitting performance while using only about 10% of the total training budget, and the authors release code for reproducibility.

Abstract

Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We then propose an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, our method consistently outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.