AI crawlers multiply fast, the documentation is scattered, and the names are easy to confuse. This is a single, current directory of the major ones. What each does, what you lose by blocking it, and how to set your robots.txt deliberately.
How to read this: the three jobs
Every bot here is doing one of three jobs (covered in depth in The AI crawler lifecycle):
- Train: gathers content that may be baked into a future model.
- Index: builds the live search index the model retrieves from and cites at answer time.
- Answer: fetches one specific page in real time because a user asked (e.g. "summarise this link").
A single operator usually runs a separate bot for each, so you can allow or block them independently.
The directory
| Operator | User-agent | Job | If you block it |
|---|---|---|---|
| OpenAI | GPTBot |
Train | Less of your content learned by future GPT models |
| OpenAI | OAI-SearchBot |
Index | You drop out of ChatGPT search citations |
| OpenAI | ChatGPT-User |
Answer | "Summarise this link" breaks for your pages |
| Anthropic | ClaudeBot |
Train | Less of your content learned by Claude |
| Anthropic | Claude-SearchBot |
Index | Absent from Claude's cited search results |
| Anthropic | Claude-User |
Answer | User-pasted-link summaries break |
Googlebot |
Index | You lose Google Search and AI Overviews / AI Mode | |
Google-Extended |
Train | Excluded from Gemini training & grounding | |
GoogleOther |
Misc / research | Minor. Research and product crawls | |
| Microsoft | Bingbot |
Index | Lose Bing and Copilot grounding |
| Perplexity | PerplexityBot |
Index | Absent from Perplexity's answers |
| Perplexity | Perplexity-User |
Answer | User-triggered fetches break |
| Apple | Applebot |
Index | Lose Siri / Spotlight surfaces |
| Apple | Applebot-Extended |
Train | Excluded from Apple Intelligence training |
| Amazon | Amazonbot |
Index | Lose Amazon's assistant surfaces |
| Meta | meta-externalagent |
Train / Index | Less presence in Meta AI |
| Meta | FacebookBot |
Index / previews | Social link previews degrade |
| ByteDance | Bytespider |
Train | Content kept out of ByteDance / TikTok AI |
| DuckDuckGo | DuckAssistBot |
Answer | Absent from DuckAssist answers |
| Cohere | cohere-ai |
Train / retrieval | Content kept out of Cohere systems |
| Mistral | MistralAI-User |
Answer | User-triggered fetches break |
| DeepSeek | DeepSeekBot |
Train | Content kept out of DeepSeek models |
| Common Crawl | CCBot |
Aggregator | See below. Closes a back-door into many models |
(Older OpenAI/Anthropic tokens like anthropic-ai and claude-web still appear in some logs; the bots above are the current set. The landscape shifts every few months. Treat this as a living list.)
The special case: Common Crawl (CCBot)
CCBot doesn't fit the three jobs. Common Crawl is a non-profit that crawls the web into a giant public archive and trains no models itself, but that archive is one of the most widely used inputs for training models across many labs.
Blocking
GPTBotandClaudeBotdoes not stop your content reaching those same labs through the Common Crawl dataset. To close that back-door, blockCCBotspecifically.
robots.txt recipes
Welcome everyone (the right default for most brands that want AI visibility):
User-agent: *
Allow: /
Allow search + answer, block training (for publishers protecting content from model training while staying visible in AI search):
# Block training crawlers
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
Disallow: /
# Everything else (search + answer bots) is welcome
User-agent: *
Allow: /
Grouping several User-agent lines above one Disallow applies that rule to all of them.
Two things robots.txt can't do
- Stop spoofing. A user-agent is self-reported. Well-behaved bots honour
robots.txt; to actually enforce access (or verify a bot is who it claims), use IP-range / reverse-DNS checks at your server or CDN. - Override your CDN. If your CDN or WAF returns a 403 to a crawler before the request reaches your site, your
robots.txtnever gets read. A silent, common failure (here's how to catch it).
Verify your own site in 30 seconds
Pretend to be a crawler and check the status code:
curl -sI -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yourdomain.com | head -1
A 200 means you're reachable; a 403 means something is blocking you regardless of robots.txt. Run it for the bots you care about most.
Deciding which of these to welcome is a strategy call. See Should you let AI crawlers index your site?. Keeping the door open to the right ones, and catching it when it silently closes, is part of what Buffy Intel monitors.