Robots.txt is a request to crawlers, not access control. The standard says its rules "are not a form of access authorization."1 Crawlers must use case-insensitive matching to find the group that matches their product token.1
The practical question is what each bot does and which token controls it.
Quick reference
| Vendor | Token | Purpose | Robots.txt behavior |
|---|---|---|---|
| OpenAI | GPTBot | Training | Disallow signals content should not be used for training2 |
| OpenAI | OAI-SearchBot | Search | Controls appearance in ChatGPT search answers2 |
| OpenAI | ChatGPT-User | User request | Rules may not apply2 |
| OpenAI | OAI-AdsBot | Ad landing page checks | Not used for training2 |
| Perplexity | PerplexityBot | Search | Not used for foundation model training3 |
| Perplexity | Perplexity-User | User request | Generally ignores robots.txt3 |
| Anthropic | ClaudeBot | Training | Honors robots.txt4 |
| Anthropic | Claude-SearchBot | Search | Honors robots.txt4 |
| Anthropic | Claude-User | User request | Honors robots.txt4 |
| Apple | Applebot | Search, and data that may feed AI | Respects standard directives5 |
| Apple | Applebot-Extended | Training opt-out control | Does not crawl5 |
| Google-Extended | Gemini training and grounding control | Standalone token6 | |
| Common Crawl | CCBot | Open web crawl repository | Disallow supported7 |
Training crawlers
GPTBot crawls content that may be used in training OpenAI's generative AI foundation models.2 ClaudeBot collects web content that could contribute to Anthropic's model training.4 Restricting it signals that the site's future materials should be excluded from training datasets.4
Two training controls are not crawlers at all. Applebot-Extended does not crawl webpages, and it is only used to determine how to use data crawled by Applebot.5 Google-Extended has no separate user agent string, and Google crawls with its existing strings while the token acts as a control.6 It manages whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI.6 It does not affect inclusion in Google Search and is not a ranking signal.6
Common Crawl is different: it is a non-profit that maintains an open repository of web crawl data.7 Blocking CCBot keeps a site out of that crawl.7 Common Crawl also warns that some crawlers falsely identify themselves as CCBot, so verify user agent strings.7
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Search crawlers
OAI-SearchBot surfaces sites in ChatGPT's search features.2 Sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links.2 OpenAI recommends allowing OAI-SearchBot in robots.txt.2 For search results, it can take about 24 hours after a robots.txt update for OpenAI's systems to adjust.2
PerplexityBot is designed to surface and link websites in Perplexity search results, and is not used to crawl content for AI foundation models.3 Claude-SearchBot improves search result quality, and disabling it may reduce a site's visibility in user search results.4
Applebot powers search in Spotlight, Siri and Safari, and allowing it lets content appear in search results for Apple users.5 Applebot crawled data may also be used to help train Apple foundation models, and disallowing Applebot-Extended opts out of that.5 Disallowing Applebot-Extended still leaves the site crawlable and discoverable through Applebot.5 If robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot instructions.5 It does not follow crawl-delay.5
Googlebot is the search crawler, and its robots.txt preferences affect Google Search and products such as Images, Video mode, News and Discover.6 Googlebot-Image preferences affect Google Images and features where images, logos and favicons are presented.6 Storebot-Google preferences affect all surfaces of Google Shopping.6 Google says its common crawlers obey robots.txt rules when crawling automatically.6
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Applebot
Allow: /
User-requested fetchers
These fetch a page because a person asked a question. ChatGPT-User visits pages for certain user actions in ChatGPT and Custom GPTs, and OpenAI says robots.txt rules may not apply because users initiate the action.2 It is not used to determine whether content may appear in Search, so OpenAI points to OAI-SearchBot for search opt-outs.2
Perplexity-User supports user actions and is not used for web crawling or training.3 Since a user requested the fetch, it generally ignores robots.txt rules.3 For Claude-User, disabling it prevents Anthropic's system from retrieving content in response to a user query.4 Anthropic states that its bots honor robots.txt directives.4
Treat these as the bots where a robots.txt line may not be enough. If a block must hold, use a server-side measure instead, such as HTTP authentication, which the robots standard names as the appropriate security approach.1
Other crawlers to know
- OAI-AdsBot visits pages submitted as ads on ChatGPT to validate safety, and its data is not used to train foundation models.2
- Google-CloudVertexBot preferences affect crawls requested by site owners for building Vertex AI Agents.6 It has no effect on Google Search or other products.6
- GoogleOther is a generic crawler for fetching public content, for example one-off internal research crawls, and does not affect a specific product.6
- iTMS crawls only URLs tied to registered Apple Podcasts content and does not follow robots.txt.5
Practical notes
Blocking by IP is a poor opt-out for Anthropic. Anthropic says blocking its IP addresses may not guarantee an opt-out because it stops the bot from reading robots.txt.4 Opt out per subdomain, since the file must sit in each host's top-level directory.4
Keep robots.txt available. If the file is unreachable because of server errors, crawlers must assume complete disallow, while a 4xx response lets them access anything.1 Anthropic supports the non-standard Crawl-delay extension.4
Combine the training and search settings deliberately. OpenAI's own example is allowing OAI-SearchBot to appear in search while disallowing GPTBot.2
To verify real bots, OpenAI publishes IP address lists for its bots.2 Apple identifies Applebot traffic by reverse DNS or a published CIDR file.5 Common Crawl runs CCBot on dedicated IP ranges with reverse DNS.7
Comments 0
No comments yet.
Sign in with GitHub or Google to comment. Reading needs no account.