Decoupling Knowledge and Task Subspaces for Composable Parametric Retrieval Augmented Generation

arXiv cs.CL / 4/30/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper focuses on a limitation of Parametric Retrieval-Augmented Generation (PRAG): document adapters trained with task-supervised objectives may entangle reusable task skills with document-specific facts, reducing stability when adapters are merged.
It proposes Orthogonal Subspace Decomposition (OSD) to disentangle these roles by training a Task LoRA for reusable task behavior and separate document LoRAs that encode knowledge in an orthogonal subspace.
The approach is designed to provide a controlled experimental setup to study how orthogonalizing task and document LoRA updates influences adapter composition in multi-document PRAG.
Experiments across multiple knowledge-intensive tasks and model scales indicate that orthogonalization improves compositional robustness, particularly when merging multiple document adapters at inference time.

Abstract

Parametric Retrieval-Augmented Generation (PRAG) encodes external documents into lightweight parameter modules that can be retrieved and merged at inference time, offering a promising alternative to in-context retrieval augmentation. Despite its potential, many PRAG implementations train document adapters with task-supervised objectives, which may cause each adapter to encode both document-specific facts and reusable task-solving behavior. This entanglement may make adapter composition less reliable: when multiple adapters are merged at inference time, their overlapping task behaviors can accumulate together with document-specific updates, potentially making the merged adapter less stable and less focused on the intended document knowledge. To examine this issue, we explore Orthogonal Subspace Decomposition (OSD), an adapter-training setup that separates reusable task behavior from document-specific knowledge adapters. Concretely, we first train a Task LoRA to capture reusable task behavior, and then train document LoRAs to encode document-specific knowledge in a orthogonal subspace. This setup provides a controlled way to examine how orthogonalizing task and document LoRA updates affects adapter composition in multi-document PRAG. Experiments across multiple knowledge-intensive tasks and model scales suggest that this orthogonalization strategy can improve compositional robustness in parametric RAG, especially when multiple document adapters are merged.