Anthropic crawler controls

Claude has separate search, training, and user agents.

Anthropic documents three robots with distinct roles. Its bots honor standard robots.txt directives, and Anthropic also documents support for the non-standard Crawl-delay directive.

Claude-SearchBot

Indexes online content to improve the relevance and accuracy of Claude search results.

ClaudeBot

Collects public web content that may contribute to future model-training datasets.

Claude-User

Retrieves content in response to a Claude user's query. Anthropic says blocking it may reduce visibility for user-directed web search.

Independent example

Allow Claude search and user retrieval while blocking model-development crawling.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Source: Anthropic crawler documentation.

Check your file