The Effect of Document Selection on Query-focused Text Analysis

arXiv cs.CL / 4/15/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper studies how document selection strategies affect query-focused text analysis outputs when analyzing only a subset of documents under compute constraints.
It systematically compares seven selection methods (from random to hybrid retrieval) across four topic/text analysis approaches (LDA, BERTopic, TopicGPT, HiCode).
Experiments are conducted on two datasets using 26 open-ended queries, allowing the authors to quantify how selection choice changes results across different analysis methods.
The findings recommend semantic or hybrid retrieval as strong default selection approaches, since weaker strategies can degrade output quality and waste compute.
By treating data selection as a methodological choice rather than a mere necessity, the work encourages the development of improved selection strategies.

Abstract

Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.