Qwen3.5-35B GGUF quants (16–22 GiB) - KLD + speed comparison

I'm back with some more benchmarks. I benchmarked the KLD divergence of the actual Qwen3.5-35B-A3B GGUF quantizations (16–22 GiB) available on Hugging Face.

KLD: The Kullback-Leibler divergence which shows how similar the FP16 and the quantized logit distributions are by measuring the difference in probability distributions between the quantized model and the FP16 baseline on a reference corpus.

u/TitwitMuffbiscuit had a shot at this some time ago but unfortunately all the models got updated a short period after he published his measurements.

For this research I also decided not to use the Wikitext-2 test dataset, which is in English, and instead took the multilingual FLORES 200 dataset out of which I extracted 700 KB of lines across randomly chosen languages. Additionally, I found another interesting dataset calibration_data_v5_rc.txt with about 400KB in size that contains a lot of interesting topics such as programming, math, syntax examples, technical text, etc. I combined both datasets into a mixed dataset to create the KLD baseline and measured the KLD distance for all the models that I found with this baseline.

I prepared two tables, where one is sorted by the classical "KLD mean" value and one that's sorted by the "KLD 99%" value, similar to the plots that Unsloth published on their latest blogpost about the Qwen models.

I'm not going to try to declare a winner here, that's up to you, given your very specific constraints as a GPU-Poor. To make it a little easier to visualize the models that are punching above their weight, i simply compare the numbers of the actual model to the model below and visualize them in bold letters if they are lower or higher based on the chosen metric.

The PP/s (prompt-processing) and TG/s (token-generation) columns are very specific numbers that will probably be meaningless to most users. You are going to need a Intel CPU, a RTX 3090 GPU (Ampere) and use Linux with Cuda Driver Version 580.126.18 to make use of those numbers. I used llama-bench with a context length of 10k to obtain these numbers.

Looking at the TG/s speed, for example, we can see that UD-Q3_K_XL from Unsloth before their last update was the slowest with a generation speed of ~105 t/s and the fastest is Mungert's iq4_nl with ~143 t/s which makes a total variation of 36.2% in the token generation speed for my specific architecture, which is shockingly high and one of the reasons why it is a little bit hard to define a so-called best model.

Notes: The cmp-nct prefixed models in the tables are actually a mirror from the older Unsloth quants that I found before their latest upload, which I also wanted to measure.

Sorted by KLD mean

Sorted by KLD 99%

Edit: If you want some models to be included that i forgot you have 24 hours to post a link to the models you want to get measured otherwise i'm going to reclaim my hdd space.