SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning

arXiv cs.CL / 4/22/2026

📰 NewsModels & Research

共有:

Key Points

The paper introduces SAMoRA, a parameter-efficient fine-tuning framework that combines Mixture-of-Experts (MoE) with LoRA for better task-adaptive multi-task learning in large language models.
It addresses two shortcomings of prior MoE-LoRA approaches: imprecise routing that does not align input semantics to expert capabilities, and uniform fusion/weighting that cannot adapt update strength to task complexity.
SAMoRA adds a Semantic-Aware Router to align textual semantics with the most suitable experts for more precise routing.
It proposes a Task-Adaptive Scaling mechanism that dynamically regulates each expert’s contribution according to task requirements.
Experiments on multiple multi-task benchmarks show SAMoRA achieves significant improvements over state-of-the-art methods and demonstrates strong task generalization, with code released on GitHub.

Abstract

The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA