OpenAI crawler controls

GPTBot and OAI-SearchBot do different jobs.

OAI-SearchBot is used to surface pages in ChatGPT search. GPTBot crawls content that may be used to improve OpenAI foundation models. OpenAI documents these as independent settings.

OAI-SearchBot

Search crawler. Blocking it opts the site out of ChatGPT search answers, though navigational links may still appear.

GPTBot

Model-development crawler. Disallowing it indicates that crawled content should not be used for training generative AI foundation models.

ChatGPT-User

User-initiated fetcher. It is not an automatic web crawler, and OpenAI says robots.txt rules may not apply.

Independent example

Allow search discovery while blocking model-development crawling.

Source: OpenAI crawler documentation. User-agent versions may change; use the product tokens in robots.txt.