Robustness Risk of Conversational Retrieval: Identifying and Mitigating Noise Sensitivity in Qwen3-Embedding Model

arXiv cs.AI / 4/10/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper empirically studies embedding-based retrieval in realistic conversational setups (short, dialogue-like, weakly specified queries) and shows that retrieval corpora can include structured conversational artifacts that act as noise.
It identifies a robustness vulnerability in Qwen3-embedding models: without query prompting, dialogue-style noise can become disproportionately retrievable and appear in top-ranked results even when it is semantically uninformative.
The failure mode is consistent across Qwen3 model scales, largely undetected by standard clean-query benchmarks, and is more pronounced for Qwen3 than for earlier Qwen variants and other common dense retrieval baselines.
The authors demonstrate that lightweight query prompting changes retrieval behavior and suppresses the noise intrusion, restoring ranking stability.
Overall, the work argues for evaluation protocols that better match deployed conversational retrieval systems to catch noise sensitivity issues.

Abstract

We present an empirical study of embedding-based retrieval under realistic conversational settings, where queries are short, dialogue-like, and weakly specified, and retrieval corpora contain structured conversational artifacts. Focusing on Qwen3-embedding models, we identify a deployment-relevant robustness vulnerability: under conversational retrieval without query prompting, structured dialogue-style noise can become disproportionately retrievable and intrude into top-ranked results, despite being semantically uninformative. This failure mode emerges consistently across model scales, remains largely invisible under standard clean-query benchmarks, and is significantly more pronounced in Qwen3 than in earlier Qwen variants and other widely used dense retrieval baselines. We further show that lightweight query prompting qualitatively alters retrieval behavior, effectively suppressing noise intrusion and restoring ranking stability. Our findings highlight an underexplored robustness risk in conversational retrieval and underscore the importance of evaluation protocols that reflect the complexities of deployed systems.