Token-Efficient Multimodal Reasoning via Image Prompt Packaging

arXiv cs.AI / 4/6/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper proposes Image Prompt Packaging (IPPg), a prompting method that embeds structured text into images to cut multimodal inference costs driven by text-token overhead.
Across five datasets, three frontier multimodal models (GPT-4.1, GPT-4o, Claude 3.5 Sonnet), and two task families (VQA and code generation), IPPg achieves reported inference cost reductions of 35.8%–91.0% with up to 96% token compression.
Accuracy effects are highly dependent on the specific model and task: GPT-4.1 shows simultaneous accuracy and cost gains on CoSQL, while Claude 3.5 increases costs on several VQA benchmarks.
The authors derive a token-type cost decomposition and provide a failure-mode taxonomy, finding spatial reasoning, non-English inputs, and character-sensitive operations to be most vulnerable, while schema-structured tasks benefit most.
A large rendering ablation (125 configurations) shows visual encoding choices materially impact results, with accuracy shifts of 10%–30%, positioning visual encoding as a key design variable for multimodal systems.

Abstract

Deploying large multimodal language models at scale is constrained by token-based inference costs, yet the cost-performance behavior of visual prompting strategies remains poorly characterized. We introduce Image Prompt Packaging (IPPg), a prompting paradigm that embeds structured text directly into images to reduce text token overhead, and benchmark it across five datasets, three frontier models (GPT-4.1, GPT-4o, Claude 3.5 Sonnet), and two task families (VQA and code generation). We derive a cost formulation decomposing savings by token type and show IPPg achieves 35.8--91.0\% inference cost reductions. Despite token compression of up to 96\%, accuracy remains competitive in many settings, though outcomes are highly model- and task-dependent: GPT-4.1 achieves simultaneous accuracy and cost gains on CoSQL, while Claude 3.5 incurs cost increases on several VQA benchmarks. Systematic error analysis yields a failure-mode taxonomy: spatial reasoning, non-English inputs, and character-sensitive operations are most vulnerable, while schema-structured tasks benefit most. A 125-configuration rendering ablation reveals accuracy shifts of 10--30 percentage points, establishing visual encoding choices as a first-class variable in multimodal system design.