FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models

arXiv cs.LG / 4/3/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

FourierMoE proposes a new parameter-efficient fine-tuning approach for LLMs by moving mixture-of-experts adaptation into the spectral (frequency) domain instead of the spatial domain used by prior MoE-PEFT methods.
The method is motivated by findings that different tasks have distinct frequency energy distributions and that LLM layers have heterogeneous sensitivities across frequencies.
FourierMoE uses a frequency-adaptive router to send tokens to experts specialized in different frequency bands, with experts learning conjugate-symmetric complex coefficients to preserve full phase and amplitude information.
The authors argue FourierMoE supports theoretically lossless reconstruction of real-valued spatial weights via inverse discrete Fourier transform (IDFT), maintaining representational fidelity during adaptation.
Experiments across 28 benchmarks, multiple model architectures, and different scales show consistent gains over baselines in both single-task and multi-task fine-tuning while requiring significantly fewer trainable parameters.

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.