OpenAI
OAI-SearchBot controls ChatGPT search inclusion. GPTBot controls model-development crawling. ChatGPT-User handles user actions.
Check your file
Paste the contents of robots.txt. This tool checks access at /; path-specific rules are shown when they affect the root.
Root-path analysis
Crawler access
Paste your file and select “Analyze access” to see the matching policy for each agent.
A robots.txt rule expresses a preference. User-initiated fetchers may bypass it, and a matching user-agent string alone does not prove a request is authentic. Follow each vendor’s verification guidance when inspecting server logs.
Source-backed definitions
Know what each policy controls
Search visibility, model development, and user-request retrieval are separate choices. The checker follows the vendors’ own current documentation instead of treating every AI bot as equivalent.
Anthropic
Claude-SearchBot, ClaudeBot, and Claude-User separately cover search, model development, and user retrieval.
Perplexity
PerplexityBot supports search results. Perplexity-User is user-initiated and generally ignores robots.txt.
Meta
meta-externalagent indexes for AI training or product improvement. User-initiated meta-externalfetcher may bypass robots.txt.
Google and Apple
Google-Extended and Applebot-Extended are usage controls, not standalone crawling bots. Blocking them does not block Googlebot or Applebot search crawling.
Common Crawl
CCBot builds Common Crawl’s public web corpus and documents a standard robots.txt opt-out.