Post-Optimization Adaptive Rank Allocation for LoRA

arXiv cs.AI / 5/1/2026

📰 NewsTools & Practical UsageModels & Research

共有:

Key Points

The paper introduces Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression approach for LoRA that adaptively assigns ranks to different layers instead of using a uniform rank everywhere.
PARA uses Singular Value Decomposition (SVD) and a single global threshold over singular values to prune LoRA ranks based on each layer’s spectral (importance) characteristics.
Because PARA is applied as a post-hoc step after standard fine-tuning, it avoids training-time modifications and the potential instability that can come with dynamic rank architectures.
Experiments on multiple vision and language benchmarks show PARA can cut LoRA parameters by 75–90% while maintaining predictive performance close to the original uncompressed LoRA.
The authors plan to release code after acceptance, aiming to make PARA easy to integrate into existing LoRA fine-tuning pipelines.

Abstract

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90\% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.