Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems

arXiv cs.CL / 4/15/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

Thought-Retriever is a new, model-agnostic algorithm designed to improve how LLM agents use very large external knowledge beyond typical context-length and top-K retrieval limits.
Instead of retrieving only raw data chunks, it stores and reuses an LLM’s intermediate “thoughts” from past interactions, filtering irrelevant content and organizing useful items in long-term thought memory.
The approach enables conditioning on arbitrarily long external data by retrieving relevant thoughts rather than relying on the model’s immediate context window.
The authors introduce a benchmark, AcademicEval, focused on faithfully leveraging ultra-long context to answer questions grounded in real academic papers.
Experiments report at least a 7.6% average F1 improvement and a 16% win-rate increase over state-of-the-art baselines, with evidence that the system self-evolves over more user queries and better uses deeper reasoning for abstract questions.

Abstract

Large language models (LLMs) have transformed AI research thanks to their powerful internal capabilities and knowledge. However, existing LLMs still fail to effectively incorporate the massive external knowledge when interacting with the world. Although retrieval-augmented LLMs are proposed to mitigate the issue, they are still fundamentally constrained by the context length of LLMs, as they can only retrieve top-K raw data chunks from the external knowledge base which often consists of millions of data chunks. Here we propose Thought-Retriever, a novel model-agnostic algorithm that helps LLMs generate output conditioned on arbitrarily long external data, without being constrained by the context length or number of retrieved data chunks. Our key insight is to let an LLM fully leverage its intermediate responses generated when solving past user queries (thoughts), filtering meaningless and redundant thoughts, organizing them in thought memory, and retrieving the relevant thoughts when addressing new queries. This effectively equips LLM-based agents with a self-evolving long-term memory that grows more capable through continuous interaction. Besides algorithmic innovation, we further meticulously prepare a novel benchmark, AcademicEval, which requires an LLM to faithfully leverage ultra-long context to answer queries based on real-world academic papers. Extensive experiments on AcademicEval and two other public datasets validate that Thought-Retriever remarkably outperforms state-of-the-art baselines, achieving an average increase of at least 7.6% in F1 score and 16% in win rate across various tasks. More importantly, we further demonstrate two exciting findings: (1) Thought-Retriever can indeed help LLM self-evolve after solving more user queries; (2) Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.