Qwen3.5-9B on document benchmarks: where it beats frontier models and where it doesn't.

We run an open document AI benchmark. 20 models, 9,000+ real documents. Just added all four Qwen3.5 sizes (0.8B to 9B). Now we have per-task breakdowns for every model.

You can see the results here : idp-leaderboard.org

Where all Qwen wins or matches:

OlmOCR (text extraction from messy scans, dense PDFs, multi-column layouts):

Qwen3.5-9B: 78.1
Qwen3.5-4B: 77.2
Gemini 3.1 Pro: 74.6
Claude Sonnet 4.6: 74.4
Qwen3.5-2B: 73.7
GPT-5.4: 73.4

9B and 4B are ahead of every frontier model on raw text extraction. The 2B matches GPT-5.4.

VQA (answering questions about document content, charts, tables):

Gemini 3.1 Pro: 85.0
Qwen3.5-9B: 79.5
GPT-5.4: 78.2
Qwen3.5-4B: 72.4
Claude Sonnet 4.6: 65.2
GPT-5.2: 63.5
Gemini 3 Flash: 63.5

This one surprised us the most. The 9B is second only to Gemini 3.1 Pro on VQA. It edges past GPT-5.4. It is 14 points ahead of Claude Sonnet and 16 points ahead of Gemini Flash. For a 9B open model, that VQA score is hard to explain.

KIE (extracting invoice numbers, dates, amounts):

Gemini 3 Flash: 91.1
Claude Opus 4.6: 89.8
Claude Sonnet 4.6: 89.5
GPT-5.2: 87.5
Gemini 3.1 Pro: 86.8
Qwen3.5-9B: 86.5
Qwen3.5-4B: 86.0
GPT-5.4: 85.7

Qwen-9B matches Gemini 3.1 Pro. Qwen-4B matches GPT-5.4. Both ahead of GPT-5-Mini (85.7), Claude Haiku (85.6), and Ministral-8B (85.7). A 4B model doing production-grade field extraction.

Where frontier models are clearly better.

Table extraction (GrITS):

Gemini 3.1 Pro: 96.4
Claude Sonnet: 96.3
GPT-5.4: 94.8
Gemini 3 Pro: 95.8
GPT-5.2: 86.0
Gemini 3 Flash: 85.6
Qwen3.5-4B: 76.7
Qwen3.5-9B: 76.6

Frontier models are 85 to 96 on tables. Qwen is stuck at 76 to 77 regardless of size. The 4B and 9B are essentially identical. This looks like an architecture limit, not a scale limit.

Handwriting OCR:

Gemini 3.1 Pro: 82.8
Gemini 3 Flash: 81.7
GPT-4.1: 75.6
Claude Opus: 74.0
Claude Sonnet: 73.7
GPT-5.4: 69.1
Ministral-8B: 67.8
Qwen3.5-9B: 65.5
Qwen3.5-4B: 64.7

Gemini dominates handwriting. Qwen is behind but not drastically behind GPT-5.4 (69.1 vs 65.5).

Scaling within the Qwen family:

Overall: 0.8B 58.0, 2B 63.2, 4B 73.1, 9B 77.0

Summary:

OCR extraction: Qwen 4B/9B ahead of all frontier models
VQA reasoning: Qwen-9B is #2 behind only Gemini 3.1 Pro. Beats GPT-5.4.
KIE field extraction: Qwen 4B/9B match frontier models
Table extraction: Frontier models lead by 10 to 20 points

Every prediction is visible. Compare Qwen outputs against any model on the same documents.

idp-leaderboard.org/explore