An AI crawler is an automated bot that fetches your web pages on behalf of an AI company. Each one identifies itself with a user-agent name. GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google).
They do one of three jobs, and the distinction matters: training crawlers gather text to teach future models, indexing/search crawlers build the index an engine searches at answer time, and live-answer fetchers pull a specific page the moment a user's question needs it. Blocking the wrong one can quietly remove you from AI answers.
You control which crawlers may access your site through robots.txt (and signal structure with an llms.txt). But access alone isn't enough. A bot that can reach a page still needs clean, structured HTML to understand and cite it.