Method
On November 12, 2026, Footnotes Lab requested https://<domain>/robots.txt and https://<domain>/llms.txt for the top 1,000 domains in Tranco list PY2NJ.1 We followed redirects, used a 12-second timeout and identified our requests as AEOFootnotesBot.1
A robots.txt counted when it returned a 200 response with a text body.1 A crawler counted as blocked when its applicable robots.txt group, its own user-agent group or the * group if no specific group applied, disallowed / without allowing /.1 An llms.txt counted when it returned a 200 response with a body starting with a Markdown H1.1
Of the 1,000 sites surveyed, 494 served a robots.txt, or 49.4% of the sample.1 The crawler shares below use those 494 sites as their denominator, not all 1,000 surveyed domains.1
Results
The table shows the number and share of sites whose applicable robots.txt rules blocked each listed agent at the root.1 Purpose labels are included so that training, search and user-request agents can be compared separately.1
| Crawler | Operator | Purpose | Sites blocking it at the root | Share of sites with a robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 107 | 21.7% |
| OAI-SearchBot | OpenAI | search | 57 | 11.5% |
| ChatGPT-User | OpenAI | user request | 66 | 13.4% |
| ClaudeBot | Anthropic | training | 105 | 21.3% |
| Claude-SearchBot | Anthropic | search | 67 | 13.6% |
| Claude-User | Anthropic | user request | 68 | 13.8% |
| PerplexityBot | Perplexity | search | 86 | 17.4% |
| Perplexity-User | Perplexity | user request | 74 | 15.0% |
| Google-Extended | training | 89 | 18.0% | |
| Applebot-Extended | Apple | training | 91 | 18.4% |
| CCBot | Common Crawl | training | 117 | 23.7% |
| Bytespider | ByteDance | training | 121 | 24.5% |
| meta-externalagent | Meta | training | 96 | 19.4% |
| Amazonbot | Amazon | training | 83 | 16.8% |
Findings
Bytespider was blocked by 121 sites, or 24.5% of sites serving robots.txt.1 CCBot was blocked by 117, or 23.7%.1 Both are classified as training agents in the survey.1
GPTBot and ClaudeBot followed, with 107 sites blocking GPTBot at the root and 105 blocking ClaudeBot.1 Their respective shares were 21.7% and 21.3%.1 These figures describe applicable root rules, not necessarily rules that named either bot explicitly.
Training and search agents
The survey counted 107 root blocks for GPTBot and 57 for OAI-SearchBot.1 The corresponding shares were 21.7% and 11.5% of sites with a robots.txt.1 OpenAI says its settings for these agents are independent, and gives the example of allowing OAI-SearchBot while disallowing GPTBot.2
The Anthropic entries show the same ordering: 105 sites blocked the training agent ClaudeBot, while 67 blocked the search agent Claude-SearchBot.1 Their shares were 21.3% and 13.6%, respectively.1 These are aggregate counts, so the comparison does not identify which individual sites used different rules for the two agents.
PerplexityBot, classified here as a search agent, had 86 root blocks, or 17.4%.1 That was a higher share than the search agents listed for OpenAI and Anthropic, but lower than the shares for several training agents.1 The table is a comparison of individual agents, not a single blocking rate for every crawler in a purpose category.
User-request agents
The survey counted 66 sites blocking ChatGPT-User at the root, 68 blocking Claude-User and 74 blocking Perplexity-User.1 Their shares were 13.4%, 13.8% and 15.0%, respectively.1 These entries sit apart from the training and search classifications in the table.
For ChatGPT-User, a root rule needs a qualification: OpenAI says the agent is used for certain user actions, and robots.txt rules may not apply because those actions are initiated by a user.2 OpenAI also says ChatGPT-User is not used to determine whether content may appear in Search.2 The survey records the applicable rule, not whether a user-initiated request was stopped.
How to read a root block
The count is based on the group applicable to an agent, not simply on whether its name appears in a file. The survey uses the agent’s own group when applicable, otherwise the * group.1 A site can therefore contribute to an agent’s block count through a general group rather than a named group.
The root measure also should not be read as a count of every restricted URL. RFC 9309 requires the most specific matching path rule to be used.3 Check the applicable group and its path rules when assessing access to a particular page, rather than assuming that the root result answers every URL-level question.
Finally, a robots.txt rule is not an access control. RFC 9309 says the protocol’s rules are not a form of access authorization.3 The figures report what the retrieved files instructed crawlers at the root, not whether requests were technically prevented.
File availability
The survey also found 116 sites serving an llms.txt under its test, or 11.6% of all sites surveyed.1 That percentage uses the full surveyed sample, unlike the crawler-block shares. Keeping the denominators separate matters when comparing file availability with the crawler results.
Comments 0
No comments yet.
Sign in with GitHub or Google to comment. Reading needs no account.