Introduction to the Controversy

Perplexity has published a response to Cloudflare’s claims that it disrespects robots.txt and engages in stealth crawling. This response comes as a result of a misunderstanding of how Perplexity’s AI assistants work and their role in fetching web content.

What are AI Assistants?

Perplexity argues that Cloudflare is mischaracterizing AI assistants as web crawlers. According to Perplexity, AI assistants should not be subject to the same restrictions as traditional web crawlers because they are user-initiated assistants. This means that they only fetch content when a user asks a specific question, and they do not store or index content ahead of time.

How Perplexity’s AI Assistants Work

Perplexity’s system fetches webpages only in response to specific user questions. For example, when a user asks for recent restaurant reviews, the assistant retrieves and summarizes relevant content on demand. This approach is different from traditional crawlers, which systematically index vast portions of the web without regard to immediate user intent.

Comparison to Google’s User-Triggered Fetches

Perplexity compares its on-demand fetching to Google’s user-triggered fetches. Although Google’s fetches are used for different purposes, such as reading text aloud or site verification, they are still an example of user-triggered fetching that bypasses robots.txt restrictions. Perplexity argues that its AI operates as an extension of a user’s request, not as an autonomous bot crawling indiscriminately.

Criticisms of Cloudflare’s Infrastructure

Perplexity also criticizes Cloudflare’s infrastructure for failing to distinguish between malicious scraping and legitimate, user-initiated traffic. The company suggests that Cloudflare’s approach to bot management risks overblocking services that are acting responsibly. Perplexity argues that a platform’s inability to differentiate between helpful AI assistants and harmful bots causes misclassification of legitimate web traffic.

Conclusion

In conclusion, Perplexity makes a strong case for the claim that Cloudflare is blocking legitimate bot traffic. The company’s decision to block Perplexity’s traffic was based on a misunderstanding of how its technology works. Perplexity’s AI assistants are designed to fetch content on demand, in response to specific user questions, and do not retain or use the fetched content for training its models. As the use of AI assistants continues to grow, it is essential to understand how they work and to develop infrastructure that can distinguish between legitimate and malicious traffic. By doing so, we can ensure that users have access to the information they need while also protecting websites from harmful bots.