Introduction to Googlebot and HTML Size Limits
Google’s John Mueller was asked about the number of megabytes of HTML that Googlebot crawls per page. The question was whether Googlebot indexes two megabytes (MB) or fifteen megabytes of data. Mueller’s answer minimized the technical aspect of the question and went straight to the heart of the issue, which is really about how much content is indexed.
GoogleBot and Other Bots
In the middle of an ongoing discussion, someone revived the question about whether Googlebot crawls and indexes 2 or 15 megabytes of data. They posted: "Hope you got whatever made you run 🙂 It would be super useful to have more precisions, and real-life examples like ‘My page is X Mb long, it gets cut after X Mb, it also loads resource A: 15Kb, resource B: 3Mb, resource B is not fully loaded, but resource A is because 15Kb’".
Panic About 2 Megabyte Limit Is Overblown
Mueller said that it’s not necessary to weigh bytes and implied that what’s ultimately important isn’t about constraining how many bytes are on a page but rather whether or not important passages are indexed. Furthermore, Mueller said that it is rare that a site exceeds two megabytes of HTML, dismissing the idea that it’s possible that a website’s content might not get indexed because it’s too big.
How Googlebot Works
Googlebot isn’t the only bot that crawls a web page. Google publishes a list of all the crawlers they use for various purposes. This means that even if Googlebot doesn’t crawl the entire page, other bots might still index the content.
How to Check if Content Passages Are Indexed
Mueller’s response confirmed a simple way to check whether or not important passages are indexed. He said: "Google has a lot of crawlers, which is why we split it. It’s extremely rare that sites run into issues in this regard, 2MB of HTML (for those focusing on Googlebot) is quite a bit. The way I usually check is to search for an important quote further down on a page – usually no need to weigh bytes."
Passages for Ranking
People have short attention spans except when they’re reading about a topic that they are passionate about. That’s when a comprehensive article may come in handy for those readers who really want to take a deep dive to learn more. From an SEO perspective, it’s essential to understand that comprehensive topic coverage is not automatically a ranking problem.
Understanding User Needs
A publisher or an SEO needs to step back and assess whether a user is satisfied with deep coverage of a topic or whether a deeper treatment of it is needed by users. There are also different levels of comprehensiveness, one with granular details and another with an overview-level of coverage of details, with links to deeper coverage. Google has long been able to rank document passages with their passage ranking algorithms.
Takeaways
While most of these takeaways aren’t represented in Mueller’s response, they do represent good practices for SEO. The key points are:
- HTML size limits belie a concern for deeper questions about content length and indexing visibility
- Megabyte thresholds are rarely a practical constraint for real-world pages
- Counting bytes is less useful than verifying whether content actually appears in search
- Searching for distinctive passages is a practical way to confirm indexing
- Comprehensiveness should be driven by user intent, not crawl assumptions
- Content usefulness and clarity matter more than document size
- User satisfaction remains the deciding factor in content performance
Conclusion
Concern over how many megabytes are a hard crawl limit for Googlebot reflects uncertainty about whether important content in a long document is being indexed and is available to rank in search. Focusing on megabytes shifts attention away from the real issues SEOs should be focusing on, which is whether the topic coverage depth best serves a user’s needs. Mueller’s response reinforces the point that web pages that are too big to be indexed are uncommon, and fixed byte limits are not a constraint that SEOs should be concerned about. By shifting their focus away from optimizing for assumed crawl limits and instead focusing on user content consumption limits, SEOs and publishers will probably have better search coverage.