So I asked myself a question (and then asked a coding model to build some pieces for me).. when we talk about the values in a layer of an LLM, how many are actually unique? The answer led me down a couple weeks of coding. (yes, with Claude, Qwen, and Gemini).
fp16 is 16 bits. most of the models I ran into really only use about 12-13 bits of unique values... but packing those into a block, we can squeeze most of the models I tried down by 10-25%. By trading a bit of inference speed for size, we can squeeze models onto smaller cards. (speed is ~ halved for my example test)
I've baked in a lossy/balanced version as well, but haven't tested it as much. What's been tested was on my small P2200 (5G) card, and CPU, and I'm working on updates for my 32G MI50.
I'm also wondering if this might be a good way to measure the "compactness" of a model.
Article is my narrative of the journey (paywall removed), and here's the current proof of concept code: https://github.com/bigattichouse/Codebook-Quantization
[link] [comments]
