Introduction to AI Bots and Website Crawlers
The internet is filled with various types of bots that crawl websites to gather information, and a recent analysis by Hostinger sheds light on the trends in AI bot traffic. After analyzing 66 billion bot requests across over 5 million websites, the study reveals that AI crawlers are taking two distinct paths. On one hand, Large Language Model (LLM) training bots are losing access to websites as more sites block them. On the other hand, AI assistant bots that power search tools like ChatGPT are expanding their reach.
Training Bots Are Getting Blocked
The analysis shows that OpenAI’s GPTBot, which collects data for model training, has seen a significant drop in website coverage from 84% to 12% over the study period. Similarly, Meta’s ExternalAgent, the largest training-category crawler by request volume, has also experienced a decline. This trend is consistent with previous studies, which found that many top news publishers block at least one training bot. The data suggests that more sites are blocking training bots, even when request volumes remain high.
Assistant Bots Tell a Different Story
In contrast to training bots, AI assistant bots that power search tools are gaining access to more websites. OpenAI’s OAI-SearchBot, which fetches content for ChatGPT’s search feature, has reached an average coverage of 55.67%. TikTok’s bot and Apple’s bot have also seen significant growth, with 25.67% and 24.33% coverage, respectively. These assistant crawls are user-triggered and more targeted, serving users directly rather than collecting training data.
Classic Search Remains Stable
Traditional search engine crawlers, such as Googlebot and Bingbot, have maintained a steady presence throughout the study. Googlebot has held an average coverage of 72% with 14.7 billion requests, while Bingbot has stayed at 57.67% coverage. This stability contrasts with the changes in the AI category, where Google’s main crawler faces a unique position since blocking it affects search visibility.
SEO Tools Show Decline
SEO and marketing crawlers have seen a decline in coverage, with Ahrefs maintaining the largest footprint at 60% coverage. However, the category overall has shrunk, which Hostinger attributes to two factors: these tools increasingly focus on sites actively doing SEO work, and website owners are blocking resource-intensive crawlers.
Why This Matters
The data confirms a pattern that’s been building over the past year: site operators are drawing a line between AI crawlers they’ll allow and those they won’t. The decision comes down to function, with training bots collecting content to improve models without sending traffic back, and assistant bots fetching content to answer specific user questions. This distinction is crucial, as it allows website owners to participate in AI search without contributing to model training.
Looking Ahead
As AI search continues to evolve, website owners must consider their strategy for allowing or blocking AI crawlers. OpenAI recommends allowing OAI-SearchBot if you want your site to appear in ChatGPT search results, even if you block GPTBot. By checking server logs and making informed decisions, website owners can balance their goals with the need to manage server load and potentially increase their AI visibility.
Conclusion
In conclusion, the study by Hostinger provides valuable insights into the trends in AI bot traffic and website crawlers. As AI search continues to grow, it’s essential for website owners to understand the differences between training bots and assistant bots and make informed decisions about which crawlers to allow or block. By doing so, they can navigate the complex landscape of AI search and ensure their website is visible to the right audience.