How often do AI crawlers ignore robots.txt? The 2026 network-scale data

At network scale, a large and growing share of AI crawlers ignore robots.txt: about 30% of AI-bot scrapes did not comply with the file's permissions in Q4 2025, up from roughly 3.3% a year earlier, according to TollBit's State of the Bots data aggregated across the publisher sites it monitors. The blunt lesson for site owners: robots.txt is a request compliant bots choose to honour, not a wall, and at web scale a meaningful slice of AI traffic no longer honours it.

This is a dated reference, not a new dataset. Last reviewed: 1 August 2026. The headline figures come from TollBit's State of the Bots reports (Q2 2025 and Q3 & Q4 2025), a bot-traffic analytics vendor whose numbers are network-wide aggregates across the publisher sites on its platform, reported via FIPP and Media Copilot. They are single-source and directional, so read the trend as firmer than any single decimal, and cite "TollBit, State of the Bots (Q4 2025)" when you reuse one. It sits alongside our controlled test of whether AI assistants respect robots.txt on a user fetch and our AI crawler share leaderboard.

How often do AI crawlers ignore robots.txt?

Increasingly often, and the trend line is steep. TollBit's network aggregate shows the share of AI-bot scrapes that bypassed robots.txt permissions climbing roughly ten-fold in a year:

Source: TollBit, State of the Bots (Q2 2025 and Q3 & Q4 2025 editions), network-wide aggregates via FIPP and Media Copilot (single-source, directional). The direction is the story: in late 2024 non-compliance was a rounding error; a year later roughly one scrape in three ignored an explicit disallow. TollBit also reported over 26 million disallowed scrapes in March 2025 alone, on sites whose robots.txt asked bots not to extract content. Treat the exact percentages as a moving aggregate, not a fixed rate, and never as a per-bot verdict.

How much AI-crawler traffic is there now?

Enough that non-compliance is no longer a fringe problem. TollBit's data shows AI-bot activity growing sharply relative to human visits and shifting toward live retrieval:

• Bot density rose fast. By Q4 2025 the ratio had tightened to about 1 AI-bot visit for every 31 human visits, roughly a 60% increase in bot density over the period.
• Retrieval overtook training. Real-time retrieval-augmented generation scrapes, the fetches that ground live answers, grew faster than training crawls and, TollBit reported, now outnumber them.
• Little of it returns a visitor. TollBit put referral traffic from AI applications at about 0.12% of visits on average, a fraction of what classic search sends, the extraction-heavy shape we cover in the crawl-to-refer ratio data.

The combined picture: more AI crawling, a rising share of it ignoring robots.txt, and very little of that reading turning into a click. Figures are TollBit network aggregates for 2025 (single-source, directional).

Why can't robots.txt stop AI crawlers?

Because robots.txt was never an enforcement mechanism, it is an honour system. The Robots Exclusion Protocol is a plain-text file that asks well-behaved bots not to fetch certain paths; it has no technical power to refuse a request. A bot that chooses to ignore it faces no barrier, and the record shows some do. TollBit also documented that a market of nearly 40 web-scraping vendors now sells access to the web, and that non-compliant scraping leans on evasion, IP rotation, user-agent spoofing, and cloud-based headless browsers, precisely the techniques that defeat a file which trusts the bot to identify itself honestly.

robots.txt is an honour system, not a lock: it tells honest bots where not to go and does nothing to the ones that ignore it, which is why a controlled test and a network aggregate can both be true at once.

That is why a self-declared user-agent string can't be trusted either. If a bot will ignore your disallow rules, it will also relabel itself, so the only reliable signal of who is really fetching you is a verified bot identity, confirmed by reverse DNS to published IP ranges, not the name in the request header.

Does this mean ChatGPT and Claude ignore robots.txt?

Not as simply as the aggregate suggests, and this is where two credible findings have to be reconciled rather than pitted against each other. Our own reference on the June-2026 user-fetch test found that when a user pasted a robots.txt-blocked URL into a chatbot, ChatGPT, Claude, and Perplexity honoured the block and declared honest user-agents; nine other assistants fetched the page anyway. TollBit's aggregate, by contrast, flags OpenAI's ChatGPT-User agent at about 42% of its requests reaching disallowed content across the sites it monitors.

These measure different things, so the 42% is not a clean "ChatGPT ignores robots.txt" headline:

• Request type differs. The controlled test is a single, explicit user-initiated fetch of one URL. The aggregate pools every request from an agent across many sites and windows, a mix of user fetches and autonomous retrieval, where the grey area of whether a user-triggered fetch should obey a crawl directive is unresolved.
• Scope and window differ. One is a point-in-time lab test on isolated canary URLs; the other is a rolling network average whose composition shifts by site configuration and quarter.
• Compliance is not a fixed property. A bot can decline one blocked URL in a test and still show a high non-compliant share in aggregate, because the two samples capture different behaviour.

The reconciled reading: the aggregate scale finding (~30% of AI scrapes non-compliant, rising) and the structural point (robots.txt can't enforce anything) are the durable takeaways. The per-agent percentages, on either side, are narrower snapshots that shouldn't be read as a bot's fixed character. Both studies point the same way, robots.txt shapes who politely stays out, not who can.

What should you do about AI-crawler non-compliance?

Decide deliberately, then enforce where enforcement actually lives. The rising non-compliance rate doesn't mean "block everything", for most brands that want AI visibility, the bigger risk is being unreadable. A short decision guide:

• Audit before you act. Check your own logs for AI-bot hits to disallowed paths before assuming you have a problem, using our companion how-to; network averages don't tell you your rate.
• Enforce at the edge for what must stay closed. For genuinely proprietary or paywalled content, use a CDN or WAF rule keyed to verified-bot identity that returns a 403, not a robots.txt line, and confirm you aren't blocking crawlers by accident.
• Open what you want cited. The crawlers that ground and cite live answers are the ones you usually want reading you; deciding whether to let AI crawlers index your site is a business call, not a default block.
• Verify identity, not headers. Group and confirm bots by published IP ranges, not the user-agent string, per our guide to seeing which AI bots crawl your site.

The deeper point is that "can I keep AI bots out?" and "am I being cited?" are different questions with different tools. robots.txt answers neither cleanly: it can't reliably keep non-compliant crawlers out, and blocking the compliant ones costs you citations. Watching who actually crawls you, whether they obey your rules, and whether that reading turns into citations, across engines and over time, is exactly what measuring AI visibility with Buffy Intel is built to do. Questions: [email protected].