Introduction to the Debate
Questions have been raised about the methodology used by the Pew Research Center in their study on Google’s AI summaries. The study’s conclusions have been challenged due to concerns over the creation of AI summaries, the sample size, and statistical reliability. This has led to a debate about the validity of the results and their implications for Google’s AI features.
Google’s Response
A spokesperson for Google has issued an official statement, arguing that the Pew research findings do not accurately reflect user interaction patterns related to AI summaries and standard search. The main points of Google’s rebuttal are:
- Users are increasingly seeking out AI features
- They’re asking more questions
- AI usage trends are increasing visibility for content creators
- The Pew research used flawed methodology
Google shared: "People are gravitating to AI-powered experiences, and AI features in Search enable people to ask even more questions, creating new opportunities for people to connect with websites. This study uses a flawed methodology and skewed queryset that is not representative of Search traffic. We consistently direct billions of clicks to websites daily and have not observed significant drops in aggregate web traffic as is being suggested."
Methodological Concerns
The sample size used in the Pew Research study has been criticized as being too low. Duane Forrester, formerly of Bing, pointed out that the sampling size of the research was too low to be meaningful, with only 900+ adults and 66,000 search queries. Forrester shared: "Out of almost 500 billion queries per month on Google and they’re extracting insights based on 0.0000134% sample size (66,000+ queries), that’s a very small sample. Not suggesting that 66,000 of something is inconsequential, but taken in the context of the volume of queries happening on any given month, day, hour or minute, it’s very technically not a rounding error and were it my study, I’d have to call out how exceedingly low the sample size is and that it may not realistically represent the real world."
Statistical Reliability
The reliability of the statistics used in the study has also been questioned. The Methodology page for the statistics used lists the following reliability scores:
- Ages 18-29: plus/minus 13.7 percentage points (low level of reliability)
- Ages 30-49: plus/minus 7.9 percentage points (moderate level of reliability)
- Ages 50-64: plus/minus 8.9 percentage points (moderate to low level of reliability)
- Age 65+: plus/minus 10.2 percentage points (low level of reliability)
These reliability scores indicate a high margin of error, making the results statistically unreliable. At best, they should be seen as rough estimates.
Comparison of Results
The Pew Research study compared search queries from different months, which is problematic. Google’s AI summaries change from month to month, and user trends may impact what gets searched. This can trigger temporary freshness updates to the search algorithms that prioritize videos and news. Comparing search results from different months is therefore not a reliable method.
Dynamic Nature of AI Search Results
AI search results are dynamic and subject to change, not just for every user but also for the same user. Searching for a query in AI Overviews and summaries can result in different links and summaries, even when using different browsers. This dynamic nature of AI search results makes it difficult to compare user queries with scraped queries from a different time period.
Examples of Dynamic AI Search Results
For example, searching for the query "What is the RLHF training in OpenAI?" using different browsers can result in different links and summaries. The screenshots below show the different links and summaries returned by Google’s AI Overviews using Vivaldi and Chrome Canary browsers.
Google AIO Via Vivaldi Browser
The links shown for the query using Vivaldi browser include Amazon Web Services, Medium, and Kili Technology.
Google AIO Via Chrome Canary Browser
The links shown for the query using Chrome Canary browser include OpenAI, Arize AI, and Hugging Face.
Impact on Publishers
The dynamic nature of AI search results can lead to inconsistent traffic for publishers. Publishers and SEOs are used to static ranking positions in search results for a given search query. However, Google’s AI Overviews and AI Mode show dynamic search results, with different links and summaries being shown for the same query. This can make it challenging for publishers to predict and optimize their content for search engines.
Conclusion
In conclusion, the methodology used by the Pew Research Center in their study on Google’s AI summaries has been challenged due to concerns over sample size, statistical reliability, and the dynamic nature of AI search results. Google has responded by arguing that the study’s findings do not accurately reflect user interaction patterns related to AI summaries and standard search. The dynamic nature of AI search results can lead to inconsistent traffic for publishers, and it is essential to consider these factors when evaluating the impact of AI on search engines.