Which AI Crawlers to Allow in robots.txt: 2026 List (GPTBot, ClaudeBot & More)

Q: Which AI crawlers should I allow in robots.txt?

There is no single correct allow list. Search agents such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot affect discovery in AI search products. GPTBot, ClaudeBot, and meta-externalagent cover model development. User-request fetchers and data-use controls behave differently again. Choose each policy based on your visibility and content-use goals.

Understanding AI Crawlers

“AI crawler” now covers several different jobs: indexing pages for AI search, collecting content for model development, retrieving a page after a user request, and expressing data-use preferences. Those jobs use separate user-agent tokens and do not have the same visibility tradeoffs.

This guide separates those roles using each vendor's current documentation. You can allow search discovery without automatically allowing model-development crawling.

Check your robots.txt before changing it

Paste your existing file into Cite.sh's browser-only checker, see the rule matched by each agent, and generate a clean policy snippet.

Open the free checker

Major AI Crawlers

GPTBot (OpenAI)

User-Agent

GPTBot

Full User-Agent String

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot (version may change)

Purpose

Crawls websites to train and improve OpenAI's models. GPTBot is the training crawler; it does not power ChatGPT search results — that's OAI-SearchBot below.

Documentation

developers.openai.com/api/docs/bots

OAI-SearchBot (OpenAI)

User-Agent

OAI-SearchBot

Full User-Agent String

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

(version may change)

Purpose

Discovers and links websites in ChatGPT's search features. If you want your site to appear (and be linked) in ChatGPT search answers, allow OAI-SearchBot. Blocking it removes you from ChatGPT search without affecting model training.

Documentation

developers.openai.com/api/docs/bots

ChatGPT-User (OpenAI)

User-Agent

ChatGPT-User

Full User-Agent String

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Purpose

Fetches live web content when a ChatGPT user asks about a specific page. This is different from GPTBot — it's for real-time, user-initiated lookups, not training.

ClaudeBot (Anthropic)

User-Agent

ClaudeBot

Purpose

Collects web content that may contribute to training Anthropic's models. Anthropic separately operates Claude-SearchBot for search indexing and Claude-User for user-requested retrieval, so each policy can be chosen independently.

Documentation

support.claude.com

PerplexityBot

User-Agent

PerplexityBot

Full User-Agent String

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Purpose

Indexes pages for Perplexity's answer engine. Perplexity actively cites sources with links, making this especially valuable. Perplexity also uses Perplexity-User for user-initiated page fetches, which generally ignores robots.txt because a human requested the page.

Documentation

docs.perplexity.ai/docs/resources/perplexity-crawlers

Meta-ExternalAgent (Meta)

User-Agent

meta-externalagent (also seen as Meta-ExternalAgent)

Purpose

Meta's crawler for use cases such as AI-model training or product improvement. Meta separately operates meta-externalfetcher for user-initiated link fetches, which may bypass robots.txt.

Documentation

developers.facebook.com

Google-Extended (Google)

User-Agent

Google-Extended

Purpose

A robots.txt control token, not a separate crawler: Google's existing crawlers honor it to decide whether your content can be used for Gemini training and grounding. Blocking Google-Extended does not affect Googlebot or your Google Search rankings.

CCBot (Common Crawl)

User-Agent

CCBot

Purpose

Common Crawl is a nonprofit that creates open web archives. Many AI models use Common Crawl data for training, including early GPT models.

robots.txt Configuration

Choose each policy based on its documented purpose. The example below allows everything, but that is not automatically the right policy for every publisher.

Allow All Documented Agents

# Example only: allow search, training, user fetches, and data-use controls
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Block Specific Sections

If you want AI access but need to protect certain areas:

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Note: Many hosting platforms and CMS systems block AI crawlers by default. Check your robots.txt to ensure you haven't inadvertently blocked them.

Complete robots.txt Example

Here's a full robots.txt optimized for AI visibility:

# Standard crawlers
User-agent: *
Allow: /

# AI Crawlers - Explicitly allowed
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

# Sitemap
Sitemap: https://www.yoursite.com/sitemap.xml

# LLM Context Files
# See https://llmstxt.org for specification

Other Controls and Fetchers

These documented agents do not all behave like automatic web crawlers:

Meta-ExternalFetcher: User-initiated Meta link retrieval. Meta says it may bypass robots.txt.
Perplexity-User: User-initiated page retrieval. Perplexity says it generally ignores robots.txt.
Applebot-Extended: A data-use control for Apple foundation-model training, not a crawler. Applebot handles crawling.
Google-Extended: A product token controlling use by Gemini apps and the Vertex AI API for Gemini, not a separate crawler.

Checking Crawler Access

To verify your robots.txt is configured correctly:

Visit yoursite.com/robots.txt directly
Use the robots.txt report in Google Search Console (Settings → robots.txt)
Check server logs for AI crawler visits
Run the Cite.sh AI Crawler Checker to inspect the major AI-specific tokens

Beyond robots.txt

To maximize AI crawler effectiveness:

Create an llm.txt file with a summary of your site
Add JSON-LD schema markup for better content understanding
Keep your sitemap.xml updated with recent lastmod dates
Ensure fast page load times (slow sites may be partially crawled)
Use semantic HTML structure

Frequently Asked Questions

Which AI crawlers should I allow in robots.txt?

There is no universal allow list. If live AI-search discovery is your goal, evaluate OAI-SearchBot, Claude-SearchBot, and PerplexityBot separately from model-development agents such as GPTBot, ClaudeBot, and meta-externalagent. User-request fetchers and data-use controls have different behavior again.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content to train OpenAI's models. OAI-SearchBot discovers and links websites for ChatGPT's search features. You can allow one without the other: blocking GPTBot keeps your content out of training data, while blocking OAI-SearchBot removes your site from ChatGPT search results.

Does blocking AI crawlers hurt my Google rankings?

No. Google Search indexing is handled by Googlebot, which is separate from the AI crawlers in this guide. Blocking Google-Extended only opts your content out of Gemini training and grounding; it does not affect how Googlebot crawls, indexes, or ranks your pages.

Is Google-Extended a real crawler?

No. Google-Extended is a robots.txt control token, not a user-agent that fetches pages. Google's existing crawlers check for it to decide whether your content may be used for Gemini AI training and grounding. You will never see "Google-Extended" itself in your server logs.

Check your own policy

See which rule wins for each documented agent and generate a policy you can review before publishing.

Run the free checker

Related Resources

For more on AI optimization, check out:

Ready to boost your AI visibility? Browse our directory of AI-optimized listings or see pricing to get started.