Web2BigTable: A Bi-Level Multi-Agent LLM System for Internet-Scale Information Search and Extraction

arXiv cs.AI / 5/1/2026

📰 NewsDeveloper Stack & InfrastructureSignals & Early TrendsModels & Research

Key Points

  • Web2BigTable is a bi-level multi-agent LLM framework designed for agentic web search that must handle both deep, single-target reasoning and broad, schema-aligned information extraction across many entities.
  • The system uses an upper-level orchestrator to decompose tasks and parallel lower-level worker agents to execute sub-problems, coordinating via a shared workspace that exposes partial findings and reduces redundant exploration.
  • It improves performance over repeated runs using a closed-loop run–verify–reflect process with persistent, human-readable external memory and self-evolving updates to each agent’s behavior.
  • Reported results show new state-of-the-art performance on WideSearch (Avg@4 success rate 38.50, and substantial gains in Row F1 and Item F1), and strong generalization to depth-oriented search on XBench-DeepSearch with 73.0 accuracy.
  • The project provides open-source code on GitHub to support adoption and further experimentation.

Abstract

Agentic web search increasingly faces two distinct demands: deep reasoning over a single target, and structured aggregation across many entities and heterogeneous sources. Current systems struggle on both fronts. Breadth-oriented tasks demand schema-aligned outputs with wide coverage and cross-entity consistency, while depth-oriented tasks require coherent reasoning over long, branching search trajectories. We introduce \textbf{Web2BigTable}, a multi-agent framework for web-to-table search that supports both regimes. Web2BigTable adopts a bi-level architecture in which an upper-level orchestrator decomposes the task into sub-problems and lower-level worker agents solve them in parallel. Through a closed-loop run--verify--reflect process, the framework jointly improves decomposition and execution over time via persistent, human-readable external memory, with self-evolving updates to each single-agent. During execution, workers coordinate through a shared workspace that makes partial findings visible, allowing them to reduce redundant exploration, reconcile conflicting evidence, and adapt to emerging coverage gaps. Web2BigTable sets a new state of the art on WideSearch, reaching an Avg@4 Success Rate of \textbf{38.50} (7.5\times the second best at 5.10), Row F1 of \textbf{63.53} (+25.03 over the second best), and Item F1 of \textbf{80.12} (+14.42 over the second best). It also generalises to depth-oriented search on XBench-DeepSearch, achieving 73.0 accuracy. Code is available at https://github.com/web2bigtable/web2bigtable.