Happy to share the latest colqwen3.5-4.5B model in the series.
ColQwen3.5-4.5B-v3 is #1 (avg) on the MTEB ViDoRe leaderboard (Pending release) at 75.67 mean, ~half the params, ~13x fewer embedding dims, ~half the memory footprint of the previous #1 model.
Thoughts: V3 edges out v2 on V3 English u@5 (0.6034 vs 0.6023), a marginal gain for substantially more compute. The real win was the V2 benchmark jump and surpassing 8B models on V3. That's where I decided to draw the line between further optimization and accepting the limitations of the model and training data.
The full evaluation trail is public, with result files covering every candidate tried.
Links:
- Models (V1, V2, V3): https://huggingface.co/athrael-soju/colqwen3.5-4.5B-v3 (Model cards may need corrections)
- All eval files are up if you want to check my homework: https://huggingface.co/datasets/athrael-soju/colqwen-optimization-trail
- Full training methodology & Case Study in the blog post: https://athrael.net/blog/research/diminishing-returns-benchmark-optimization
- Mteb Leaderboard (Select ViDoRe V3 from the sidebar on the left): https://huggingface.co/spaces/mteb/leaderboard
ColQwen3.5-4.5B-v3 is already officially supported by colpali-engine and vLLM (ROCm + CUDA), so you can actually use the thing.
License: Apache 2.0
I'm now training the 9B variant with a much simpler setup and will post once that's done.
[link] [comments]
