VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

arXiv cs.CL / 3/18/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

VQKV introduces a training-free vector-quantization approach to compress the KV cache in LLMs, enabling high compression without retraining.
The method achieves an 82.8% compression ratio on LLaMA3.1-8B while preserving 98.6% of baseline performance on the LongBench benchmark.
It enables about 4.3x longer generation length at the same memory footprint, extending context capacity in resource-constrained environments.
By representing thousands of floating-point KV values with a small set of integer indices, VQKV dramatically reduces memory usage for deployments in limited hardware.

Abstract

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8\% compression ratio on LLaMA3.1-8B while retaining 98.6\% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.