Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight

arXiv cs.CL / 4/13/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper argues that in-context learning task “demonstrations” can be represented by task vectors (TVs), but prior extraction methods were cumbersome and didn’t explain how TVs actually affect computation.
It introduces Learned Task Vectors (LTVs) trained directly rather than extracted from hidden states/outputs, reporting improved accuracy, flexibility across layers/positions, and compatibility with in-context learning prompts.
Through mechanistic analysis, the authors show TVs influence predictions mainly via attention-head OV (output projection) circuits, with a small set of “key heads” driving most of the effect.
They further find that despite transformer nonlinearities, TV propagation is largely linear across the network, where early TVs rotate into task-relevant subspaces to improve relevant label logits, and later TVs mainly adjust (scale) magnitude.

Abstract

Large Language Models (LLMs) can perform new tasks from in-context demonstrations, a phenomenon known as in-context learning (ICL). Recent work suggests that these demonstrations are compressed into task vectors (TVs), compact task representations that LLMs exploit for predictions. However, prior studies typically extract TVs from model outputs or hidden states using cumbersome and opaque methods, and they rarely elucidate the mechanisms by which TVs influence computation. In this work, we address both limitations. First, we propose directly training Learned Task Vectors (LTVs), which surpass extracted TVs in accuracy and exhibit superior flexibility-acting effectively at arbitrary layers, positions, and even with ICL prompts. Second, through systematic analysis, we investigate the mechanistic role of TVs, showing that at the low level they steer predictions primarily through attention-head OV circuits, with a small subset of "key heads" most decisive. At a higher level, we find that despite Transformer nonlinearities, TV propagation is largely linear: early TVs are rotated toward task-relevant subspaces to improve logits of relevant labels, while later TVs are predominantly scaled in magnitude. Taken together, LTVs not only provide a practical approach for obtaining effective TVs but also offer a principled lens into the mechanistic foundations of ICL.