Introduction to AI Agents and Website Traffic
A Google engineer has warned that AI agents and automated bots will soon flood the internet with traffic. Gary Illyes, who works on Google’s Search Relations team, stated that “everyone and my grandmother is launching a crawler” during a recent podcast. This warning comes from Google’s latest Search Off the Record podcast episode.
The Impact of AI Agents on Websites
Illyes warned that AI agents and “AI shenanigans” will be significant sources of new web traffic. He explained that the web is getting congested, but it’s not something that the web cannot handle, as it’s designed to handle automatic traffic. This surge occurs as businesses deploy AI tools for content creation, competitor research, market analysis, and data gathering. Each tool requires crawling websites to function, and with the rapid growth of AI adoption, this traffic is expected to increase.
How Google’s Crawler System Works
The podcast provides a detailed discussion of Google’s crawling setup. Rather than employing different crawlers for each product, Google has developed one unified system. Google Search, AdSense, Gmail, and other products utilize the same crawler infrastructure. Each one identifies itself with a different user agent name, but all adhere to the same protocols for robots.txt and server health. Illyes explained that you can fetch with it from the internet, but you have to specify your own user agent string. This unified approach ensures that all Google crawlers adhere to the same protocols and scale back when websites encounter difficulties.
The Real Resource Hog: It’s Not Crawling
Illyes challenged conventional SEO wisdom with a potentially controversial claim: crawling doesn’t consume significant resources. He stated that it’s not crawling that is eating up the resources, it’s indexing and potentially serving or what you are doing with the data. He even joked he would “get yelled at on the internet” for saying this. This perspective suggests that fetching pages uses minimal resources compared to processing and storing the data. For those concerned about crawl budget, this could change optimization priorities.
The Growth of the Web
The Googlers provided historical context. In 1994, the World Wide Web Worm search engine indexed only 110,000 pages, whereas WebCrawler managed to index 2 million. Today, individual websites can exceed millions of pages. This rapid growth necessitated technological evolution. Crawlers progressed from basic HTTP 1.1 protocols to modern HTTP/2 for faster connections, with HTTP/3 support on the horizon.
Google’s Efficiency Battle
Google spent last year trying to reduce its crawling footprint, acknowledging the burden on site owners. However, new challenges continue to arise. Illyes explained the dilemma: “You saved seven bytes from each request that you make and then this new product will add back eight.” Every efficiency gain is offset by new AI products requiring more data. This is a cycle that shows no signs of stopping.
Preparing for the Traffic Surge
The upcoming traffic surge necessitates action in several areas:
- Infrastructure: Current hosting may not support the expected load. Assess server capacity, CDN options, and response times before the influx occurs.
- Access Control: Review robots.txt rules to control which AI crawlers can access your site. Block unnecessary bots while allowing legitimate ones to function properly.
- Database Performance: Illyes specifically pointed out “expensive database calls” as problematic. Optimize queries and implement caching to alleviate server strain.
- Monitoring: Differentiate between legitimate crawlers, AI agents, and malicious bots through thorough log analysis and performance tracking.
The Path Forward
Illyes pointed to Common Crawl as a potential model, which crawls once and shares data publicly, reducing redundant traffic. Similar collaborative solutions may emerge as the web adapts. While Illyes expressed confidence in the web’s ability to manage increased traffic, the message is clear: AI agents are arriving in massive numbers. Websites that strengthen their infrastructure now will be better equipped to weather the storm. Those who wait may find themselves overwhelmed when the full force of the wave hits.
Conclusion
In conclusion, the rise of AI agents and automated bots will significantly impact website traffic. Google’s engineer, Gary Illyes, has warned that the web is getting congested, but it’s designed to handle automatic traffic. To prepare for the traffic surge, website owners should assess their infrastructure, review access control, optimize database performance, and monitor traf