AEO Footnotes

Loading search…

ChatGPT search bots: What OAI-SearchBot, GPTBot and ChatGPT-User do

Use OAI-SearchBot to manage search access, GPTBot for training preferences, and ChatGPT-User to identify user-initiated visits.

Footnotes DeskAI search, AI discovery3 min read

Drafted with AI assistance from the sources in its footnotes and checked against them before publishing.

Start with the search setting

OAI-SearchBot is the agent OpenAI uses to surface websites in ChatGPT search results.1 Sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, although they can still appear as navigational links.1 For a publisher deciding whether to permit search access, that distinction is the starting point.

Allowing the search bot is not the same decision as allowing a training crawler. OpenAI says the OAI-SearchBot and GPTBot settings are independent, so a site can allow the former while disallowing the latter.1 Decide those policies separately, then check that the deployed robots.txt reflects both choices.

The agents and their jobs

Use the agent name, rather than a general label such as “AI bot,” when reviewing access rules or request logs.

The distinction matters when interpreting a visit. A ChatGPT-User request is associated with a user action, not an automatic web crawl, according to OpenAI.1 Do not treat that agent’s appearance in logs as a substitute for checking the OAI-SearchBot setting.

OpenAI also says that, because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply.1 If the goal is to opt out of search, use the OAI-SearchBot rule rather than relying on a rule for ChatGPT-User.1

Set search and training preferences separately

For search visibility, OpenAI recommends allowing OAI-SearchBot in the site’s robots.txt and allowing requests from its published IP ranges.1 For training, disallowing GPTBot signals that the site’s content should not be used to train OpenAI’s generative AI foundation models.1 Those are different settings for different purposes.

A practical review should therefore start with the intended outcome. If the site should be eligible to appear in search answers but the publisher wants to disallow training use, check that OAI-SearchBot is allowed and GPTBot is disallowed. If the publisher wants to opt out of search answers, check the OAI-SearchBot rule specifically.

If a site allows both bots, OpenAI may use the results of just one crawl for both use cases to avoid duplicate crawling.1 When reviewing logs, keep that possibility in mind rather than assuming each permitted use will produce a separate crawl.

For search results, OpenAI says its systems can take approximately 24 hours to adjust after a robots.txt update.1 After a change, allow for that adjustment period before evaluating the effect. Keep the timing tied to the search setting: it is not a general timetable for every outcome associated with every agent.

Check requests and published IP ranges

Use the relevant published list when checking whether infrastructure settings allow the requests the site intends to permit:

  • OAI-SearchBot: https://openai.com/searchbot.json.1
  • GPTBot: https://openai.com/gptbot.json.1
  • OpenAI publishes the ChatGPT-User IP addresses at https://openai.com/chatgpt-user.json.1

When fetching robots.txt, OAI-SearchBot may use a user-agent string with an additional robots.txt marker.1 OpenAI may also add that marker to GPTBot’s user-agent string when fetching robots.txt.1 The marker helps site owners distinguish requests for robots.txt from requests for other resources, particularly when logs do not include paths.1 Check the full user-agent string when separating policy-file requests from other visits.

Keep the purpose of each check clear. An IP access review addresses whether requests can reach the site, while the robots.txt review records the publisher’s bot-specific preferences. Neither should replace the other in a search-access audit.

Separate eligibility from selection

The documented OAI-SearchBot rule gives publishers a clear search opt-out and a way to allow access. It does not turn an access decision into a promise that a particular page will appear in an answer. Treat search eligibility and the selection of a page for an individual answer as separate questions.

When assessing an answer, inspect what is actually shown and keep that observation separate from the site’s bot policy. A permitted crawler setting is useful context for the audit, not an explanation of why a page was selected for a particular response.

Footnotes 1

  1. 1

Comments 0

No comments yet.

Sign in with GitHub or Google to comment. Reading needs no account.