AEO Footnotes

Loading search…

Which AI crawlers the Tranco top 1,000 block in robots.txt

Footnotes Lab measured root-level blocks across training, search and user-request agents on sites that serve a robots.txt.

Footnotes LabAI discovery, Experiments4 min read

Drafted with AI assistance from the sources in its footnotes and checked against them before publishing.

Method

On November 12, 2026, Footnotes Lab requested https://<domain>/robots.txt and https://<domain>/llms.txt for the top 1,000 domains in Tranco list PY2NJ.1 We followed redirects, used a 12-second timeout and identified our requests as AEOFootnotesBot.1

A robots.txt counted when it returned a 200 response with a text body.1 A crawler counted as blocked when its applicable robots.txt group, its own user-agent group or the * group if no specific group applied, disallowed / without allowing /.1 An llms.txt counted when it returned a 200 response with a body starting with a Markdown H1.1

Of the 1,000 sites surveyed, 494 served a robots.txt, or 49.4% of the sample.1 The crawler shares below use those 494 sites as their denominator, not all 1,000 surveyed domains.1

Results

The table shows the number and share of sites whose applicable robots.txt rules blocked each listed agent at the root.1 Purpose labels are included so that training, search and user-request agents can be compared separately.1

CrawlerOperatorPurposeSites blocking it at the rootShare of sites with a robots.txt
GPTBotOpenAItraining10721.7%
OAI-SearchBotOpenAIsearch5711.5%
ChatGPT-UserOpenAIuser request6613.4%
ClaudeBotAnthropictraining10521.3%
Claude-SearchBotAnthropicsearch6713.6%
Claude-UserAnthropicuser request6813.8%
PerplexityBotPerplexitysearch8617.4%
Perplexity-UserPerplexityuser request7415.0%
Google-ExtendedGoogletraining8918.0%
Applebot-ExtendedAppletraining9118.4%
CCBotCommon Crawltraining11723.7%
BytespiderByteDancetraining12124.5%
meta-externalagentMetatraining9619.4%
AmazonbotAmazontraining8316.8%

Findings

Bytespider was blocked by 121 sites, or 24.5% of sites serving robots.txt.1 CCBot was blocked by 117, or 23.7%.1 Both are classified as training agents in the survey.1

GPTBot and ClaudeBot followed, with 107 sites blocking GPTBot at the root and 105 blocking ClaudeBot.1 Their respective shares were 21.7% and 21.3%.1 These figures describe applicable root rules, not necessarily rules that named either bot explicitly.

Training and search agents

The survey counted 107 root blocks for GPTBot and 57 for OAI-SearchBot.1 The corresponding shares were 21.7% and 11.5% of sites with a robots.txt.1 OpenAI says its settings for these agents are independent, and gives the example of allowing OAI-SearchBot while disallowing GPTBot.2

The Anthropic entries show the same ordering: 105 sites blocked the training agent ClaudeBot, while 67 blocked the search agent Claude-SearchBot.1 Their shares were 21.3% and 13.6%, respectively.1 These are aggregate counts, so the comparison does not identify which individual sites used different rules for the two agents.

PerplexityBot, classified here as a search agent, had 86 root blocks, or 17.4%.1 That was a higher share than the search agents listed for OpenAI and Anthropic, but lower than the shares for several training agents.1 The table is a comparison of individual agents, not a single blocking rate for every crawler in a purpose category.

User-request agents

The survey counted 66 sites blocking ChatGPT-User at the root, 68 blocking Claude-User and 74 blocking Perplexity-User.1 Their shares were 13.4%, 13.8% and 15.0%, respectively.1 These entries sit apart from the training and search classifications in the table.

For ChatGPT-User, a root rule needs a qualification: OpenAI says the agent is used for certain user actions, and robots.txt rules may not apply because those actions are initiated by a user.2 OpenAI also says ChatGPT-User is not used to determine whether content may appear in Search.2 The survey records the applicable rule, not whether a user-initiated request was stopped.

How to read a root block

The count is based on the group applicable to an agent, not simply on whether its name appears in a file. The survey uses the agent’s own group when applicable, otherwise the * group.1 A site can therefore contribute to an agent’s block count through a general group rather than a named group.

The root measure also should not be read as a count of every restricted URL. RFC 9309 requires the most specific matching path rule to be used.3 Check the applicable group and its path rules when assessing access to a particular page, rather than assuming that the root result answers every URL-level question.

Finally, a robots.txt rule is not an access control. RFC 9309 says the protocol’s rules are not a form of access authorization.3 The figures report what the retrieved files instructed crawlers at the root, not whether requests were technically prevented.

File availability

The survey also found 116 sites serving an llms.txt under its test, or 11.6% of all sites surveyed.1 That percentage uses the full surveyed sample, unlike the crawler-block shares. Keeping the denominators separate matters when comparing file availability with the crawler results.

Footnotes 3

  1. 1
  2. 2
  3. 3

Comments 0

No comments yet.

Sign in with GitHub or Google to comment. Reading needs no account.

Which AI crawlers the Tranco top 1,000 block in robots.txt | AEO Footnotes