Introduction to LLMs.txt
Google’s John Mueller has shared his thoughts on LLMs.txt, a proposed standard for showing website content to AI agents and crawlers. He downplayed its usefulness, comparing it to the keywords meta tag, which is no longer used by search engines. This comparison is based on the idea that LLMs.txt is not a reliable way to control how AI bots crawl a website.
What is LLMs.txt?
LLMs.txt is often compared to robots.txt, but this comparison is incorrect. While robots.txt is used to control how bots crawl a website, LLMs.txt is used to show content to large language models (LLMs) in a specific format. This format uses markdown language, which is human and machine-readable, to indicate headings, lists, and other content elements. The purpose of LLMs.txt is to provide a simplified version of a web page’s content, without advertising and navigation, to AI bots.
Key Points About LLMs.txt
- LLMs.txt is not a way to control AI bots.
- LLMs.txt is a way to show the main content to AI bots.
- LLMs.txt is just a proposal and not a widely used and accepted standard.
The Usefulness of LLMs.txt
John Mueller’s comments on LLMs.txt suggest that it is not a useful standard. He pointed out that none of the major AI services, including Google, have announced support for LLMs.txt. This means that even if a website uses LLMs.txt, it is unlikely to have any impact on how AI bots crawl and index the site. Mueller also noted that using LLMs.txt could be redundant, as AI bots can already crawl and index website content directly.
Comparison to Keywords Meta Tag
Mueller compared LLMs.txt to the keywords meta tag, which was once used to tell search engines what keywords a website was relevant for. However, this tag was eventually abandoned because it was too easy for website owners to spam and manipulate. Similarly, LLMs.txt could be used to show one set of content to AI bots and another set to users and search engines, which could lead to spam and cloaking.
Conclusion
In conclusion, LLMs.txt is a proposed standard for showing website content to AI agents and crawlers, but its usefulness is limited. John Mueller’s comments suggest that it is not a reliable way to control how AI bots crawl a website, and it may be redundant or even lead to spam. As a result, website owners and SEOs should be cautious when using LLMs.txt and focus on creating high-quality, relevant content that can be crawled and indexed by AI bots and search engines directly.