When a user pastes a URL into a chatbot, does the assistant check your robots.txt before fetching it? A June-2026 test by Search Engine World put 12 assistants to the test, and only three. ChatGPT, Claude, and Perplexity. Honoured a robots.txt block; the other nine fetched the page anyway. The lesson for site owners is blunt: robots.txt is a request that compliant bots choose to honour, not a wall, so it shapes who politely stays out, not who can.
This is part of the crawlers and technical series, and a companion to the test on whether AI assistants render your JavaScript. Here the question is compliance: does the assistant respect the file that's supposed to govern access?
What did the robots.txt test measure?
On-demand fetches, verified by logs. For each assistant the researchers built two isolated test URLs. One allowed, one disallowed in robots.txt. Each carrying a unique "canary" reference string. They pasted the blocked URL into each assistant's chat and asked it to read the page, then checked their own server logs (not the chatbot's self-report) to see whether the fetch happened and whether the page returned its secret string. Every page that was fetched returned its exact canary, which proves a genuine retrieval rather than a hallucinated answer.
One scope note to read the result honestly: this tests user-initiated fetches: a person explicitly asking the assistant to open a specific URL. That's the ChatGPT-User / Claude-User category of activity, which several vendors classify as a user action rather than autonomous crawling, and the Robots Exclusion Protocol is genuinely ambiguous about whether it should apply. The test doesn't settle that debate; it shows what the assistants do.
Which assistants respected robots.txt, and which ignored it?
Only a quarter complied. The three that did also declared honest, identifiable user-agents; the nine that didn't used a range of disguises.
| Behaviour | Assistants | User-agent |
|---|---|---|
| Respected the block | ChatGPT, Claude, Perplexity | Honest, declared (ChatGPT-User, Claude-User, Perplexity-User) |
| Ignored the block | Gemini, Meta AI, Microsoft Copilot, Grok, DeepSeek, Qwen, ERNIE, Kimi | Disguised or undeclared |
Among the nine non-compliant systems, the disguises escalated: Gemini fetched as a generic Google client, Meta AI used undeclared crawler variants, Copilot outsourced the fetch to Diffbot, and several (DeepSeek, Qwen, Kimi) presented faked browser identities. Including "proxy swarms" across multiple countries with impossible user-agent strings like Windows NT 11.0. All findings are attributed to Search Engine World's June-2026 test; it is a single experiment focused on user-initiated fetches, so treat the specific pass/fail list as a point-in-time snapshot that will shift as vendors change behaviour.
Why doesn't robots.txt reliably stop AI fetchers?
Because it was never an enforcement mechanism. The Robots Exclusion Protocol is a published request: a well-behaved AI crawler reads your robots.txt and voluntarily stays out of disallowed paths. Nothing in the protocol forces compliance. A fetcher that decides to ignore it, or that classifies a user-pasted URL as a user action outside the protocol's scope. Simply requests the page, and your server returns it.
robots.txt tells polite bots where not to go. It does nothing to a bot that doesn't ask politely. If a page must stay private, enforce that at the server, not in a text file the fetcher is free to ignore.
That reframes two common mistakes. First, using robots.txt as a security control: it is the wrong tool, because the bots you most want to keep out are the least likely to honour it. Second, assuming a Disallow line guarantees you won't appear in an answer. It doesn't, if the assistant fetches anyway. To genuinely block traffic you need a CDN or WAF rule that returns a 403 by user-agent, IP, or behaviour. The same layer that, configured wrong, ends up blocking AI crawlers by accident.
What should I actually do about it?
Decide your stance deliberately, then enforce it where enforcement lives, and verify with logs:
- If you want AI visibility (most brands): allow them. Don't rely on robots.txt to block, and don't block by accident. Confirm the major user-agents in the AI crawler user-agent directory aren't disallowed. The cost of being unreadable is being uncited.
- If you must protect content: enforce at the CDN/WAF. A robots.txt
Disallowis a courtesy, not a lock. Put paywalled or proprietary content behind real access control and return a 403 to fetchers you've decided to exclude. - Identify traffic by behaviour, not just the declared name. Because non-compliant fetchers disguise their user-agent, read your raw server logs to see which AI bots actually crawl your site. Disguised fetches hide behind ordinary browser strings and won't show up in a name-only filter.
- Don't trust the chatbot's word. Compliance is what your logs show, not what the assistant claims when asked.
robots.txt remains worth getting right. It's how you signal intent to the bots that honour it, and how you avoid the far more common own-goal of blocking the crawlers that would have cited you. But treat it as a signal, not a guarantee. Knowing which engines reach your pages, under which user-agents, and whether that turns into citations is exactly what Buffy Intel keeps watch on. Questions: [email protected].