Method
On 2026-10-29 we requested https:///robots.txt and https:///llms.txt for the 118 distinct domains of the published tools and agencies in the AEO Footnotes directory.1 We followed redirects, used a 12 second timeout and the user agent AEOFootnotesBot.1 A robots.txt counts when it answers 200 with a text body.1 An llms.txt counts when it answers 200 with a body that starts with a markdown H1.1
A crawler counts as blocked when the robots.txt group that applies to it under RFC 9309 disallows / without allowing /.1 The applicable group is the crawler's own user-agent group, else the * group.1 Shares in the crawler table are of the sites that serve a robots.txt.1
Results
| Crawler | Operator | Purpose | Sites blocking it at the root | Share of sites with a robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | training | 1 | 0.9% |
| OAI-SearchBot | OpenAI | search | 0 | 0.0% |
| ChatGPT-User | OpenAI | user request | 0 | 0.0% |
| ClaudeBot | Anthropic | training | 1 | 0.9% |
| Claude-SearchBot | Anthropic | search | 0 | 0.0% |
| Claude-User | Anthropic | user request | 0 | 0.0% |
| PerplexityBot | Perplexity | search | 1 | 0.9% |
| Perplexity-User | Perplexity | user request | 0 | 0.0% |
| Google-Extended | training | 1 | 0.9% | |
| Applebot-Extended | Apple | training | 1 | 0.9% |
| CCBot | Common Crawl | training | 1 | 0.9% |
| Bytespider | ByteDance | training | 1 | 0.9% |
| meta-externalagent | Meta | training | 0 | 0.0% |
| Amazonbot | Amazon | training | 0 | 0.0% |
Findings
robots.txt is close to universal
Sites serving a robots.txt: 116, which is 98.3% of sites surveyed.1 Crawler shares in the table are of those sites that serve a robots.txt.1
Root-level blocking of AI crawlers is rare
Every crawler in the table is blocked at the root by at most one site.1 Seven crawlers are blocked by exactly one site each: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider.1 Each of those is 0.9% of sites with a robots.txt.1
Seven crawlers are blocked at the root by no site: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Perplexity-User, meta-externalagent and Amazonbot.1
Training, search and user-request crawlers
Among training crawlers, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider are each blocked by one site.1 Meta's meta-externalagent and Amazon's Amazonbot are blocked by none.1 Among search crawlers, PerplexityBot is blocked by one site, while OAI-SearchBot and Claude-SearchBot are blocked by none.1 All three user-request agents in the data, ChatGPT-User, Claude-User and Perplexity-User, are blocked by none.1
The data records only counts per crawler, so we make no claim about whether the blocking site is the same across crawlers.
A note on what "blocked" covers
Our count reflects a root-level disallow in the group that applies to each crawler.1 OpenAI's documentation says each of its settings is independent of the others.2 OpenAI also says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply.2 A zero for a user-request agent in our table means no root-level disallow was found, which is a statement about files and not about fetch behavior.
llms.txt is present on just over half of sites
Sites serving an llms.txt: 64, which is 54.2% of sites surveyed.1 The share serving a robots.txt was 98.3% of sites surveyed.1
The llms.txt proposal says the H1 with the name of the project or site is the only required section.3 That is why our test checks for a markdown H1 at the start of the body.1 The proposal also says the file can be placed at the site root, or at any path within it.3 Our survey requested the root path, so a file at a subpath such as /docs/ would not be counted.1
How the two files differ in purpose
The llms.txt proposal says robots.txt lets automated tools know what access to a site is considered acceptable.3 It says llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.3 RFC 9309 states that its rules are not a form of access authorization.4
What to check on your own site
These are practical steps, not findings from the survey.
- Fetch /robots.txt and confirm it answers 200 with a text body.
- Read which group applies to each AI crawler you care about. If there is no group for it, the * group applies.
- Decide training, search and user-request crawlers separately, since operators list them as separate user agents.
- Fetch /llms.txt and confirm the body starts with a markdown H1.
Limits
The survey covers 118 domains of published tools and agencies in one directory on one date.1 It tests root-level disallows only, and it does not test whether any crawler obeys a given file.1
Comments 0
No comments yet.
Sign in with GitHub or Google to comment. Reading needs no account.