When to Think and When to Look: Uncertainty-Guided Lookback

arXiv cs.CL / 3/30/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper provides a first large-scale, controlled study of how explicit test-time “thinking” (reasoning chains) affects visual reasoning performance in large vision-language models on MMMU-val.
Results show that more thinking is not always beneficial: long reasoning chains can lead models down incorrect paths, sometimes even underperforming standard instruct-mode decoding that focuses on the image.
The authors find that certain short “lookback” phrases that explicitly refer back to the image are enriched in successful trajectories and correlate with better visual grounding.
They introduce “uncertainty guided lookback,” a training-free decoding strategy that uses uncertainty signals plus adaptive lookback prompts and breadth search to improve performance.
The method yields overall MMMU gains, especially in categories where standard thinking is weak, exceeds multiple decoding baselines, and generalizes to five additional benchmarks with consistent improvements.

Abstract

Test-time thinking (that is, generating explicit intermediate reasoning chains) is known to boost performance in large language models and has recently shown strong gains for large vision language models (LVLMs). However, despite these promising results, there is still no systematic analysis of how thinking actually affects visual reasoning. We provide the first such analysis with a large scale, controlled comparison of thinking for LVLMs, evaluating ten variants from the InternVL3.5 and Qwen3-VL families on MMMU-val under generous token budgets and multi pass decoding. We show that more thinking is not always better; long chains often yield long wrong trajectories that ignore the image and underperform the same models run in standard instruct mode. A deeper analysis reveals that certain short lookback phrases, which explicitly refer back to the image, are strongly enriched in successful trajectories and correlate with better visual grounding. Building on this insight, we propose uncertainty guided lookback, a training free decoding strategy that combines an uncertainty signal with adaptive lookback prompts and breadth search. Our method improves overall MMMU performance, delivers the largest gains in categories where standard thinking is weak, and outperforms several strong decoding baselines, setting a new state of the art under fixed model families and token budgets. We further show that this decoding strategy generalizes, yielding consistent improvements on five additional benchmarks, including two broad multimodal suites and math focused visual reasoning datasets.