How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews

arXiv cs.CL / 5/1/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisIndustry & Market MovesModels & Research

共有:

Key Points

The study examines how generative AI changes web search by comparing Google Search, Google’s AI Overviews (AIO), and Gemini Flash 2.5 across 11,500 real user queries.
AIOs are produced for 51.5% of real-user queries and are commonly shown above organic results, with controversial questions especially likely to trigger an AIO.
Retrieved sources differ sharply across systems (less than 0.2 average Jaccard similarity), with traditional Google favoring popular/institutional government or education sites and generative engines more often selecting Google-owned content.
Sites blocking Google’s AI crawler are far less likely to be retrieved by AIOs, even if they otherwise contain accessible content.
AIO responses are less stable across repeated runs of the same query and more sensitive to small query edits, raising concerns about consistency and robustness in generative search.
The authors argue this has major implications for website visibility, the effectiveness of “generative engine optimization” tactics, and information users receive, calling for revenue frameworks to support a sustainable publisher–generative search ecosystem.

Abstract

Generative AI is being increasingly integrated into web search for the convenience it provides users. In this work, we aim to understand how generative AI disrupts web search by retrieving and presenting the information and sources differently from traditional search engines. We introduce a public benchmark dataset of 11,500 user queries to support our study and future research of generative search. We compare the search results returned by Google's search engine, the accompanying AI Overview (AIO), and Gemini Flash 2.5 for each query. We have made several key findings. First, we find that for 51.5\% of representative, real-user queries, AIOs are generated, and are displayed above the organic search results. Controversial questions frequently result in an AIO. Second, we show that the retrieved sources are substantially different for each search engine (<0.2 average Jaccard similarity). Traditional Google search is significantly more likely to retrieve information from popular or institutional websites in government or education, while generative search engines are significantly more likely to retrieve Google-owned content. Third, we observe that websites that block Google's AI crawler are significantly less likely to be retrieved by AIOs, despite having access to the content. Finally, AIOs are less consistent when processing two runs of the same query, and are less robust to minor query edits. Our findings have important implications for understanding how generative search impacts website visibility, the effectiveness of generative engine optimization techniques, and the information users receive. We call for revenue frameworks to foster a sustainable and mutually beneficial ecosystem for publishers and generative search providers.