How do I know an AI bot ignored my robots.txt versus just crawling allowed pages?

Compare each request against what your robots.txt disallows for that user-agent. A non-compliant hit is an HTTP 200 response to a path you disallowed for that bot. If the bot only fetched allowed paths, it complied, even if it crawled heavily. So you need two things side by side: your exact disallow rules per user-agent, and the list of paths each AI bot actually fetched. The overlap, blocked paths that returned content, is your non-compliance.

The user-agent says GPTBot, but can I trust it?

No. A user-agent string is self-declared and trivially spoofed, and non-compliant scrapers routinely fake identities or rotate IPs. Before you count a request as a real bot, verify it against the operator's published IP ranges (a reverse-DNS or verified-bot check). Requests claiming to be a named bot from an unverifiable IP are their own bucket, likely impersonation, and are a different problem from a real bot ignoring your rules.

Should I block the bots that ignore my robots.txt?

Only for content you genuinely need closed, and only at the edge. For proprietary or paywalled paths, a CDN or WAF 403 keyed to verified identity actually enforces the boundary that robots.txt can't. But for most pages, the crawlers ignoring your file are the ones grounding AI answers, and blocking them forfeits citations. Decide path by path: enforce what must stay private, and let the rest be read.

How to check whether AI bots are ignoring your robots.txt

Network data suggests about 30% of AI-bot scrapes ignore robots.txt, but that is a web-wide average, not your number. The only way to know whether AI bots respect your robots.txt is to compare their requests against your own disallow rules in your logs. This is the hands-on companion to how often AI crawlers ignore robots.txt; read that for the benchmark figures. Below is the method, in five steps.

Step 1: Write down exactly what your robots.txt disallows

You can't detect a violation without a rule to check against. Open your live robots.txt and list, per user-agent, every path you disallow. Robots directives are user-agent-specific, so a path blocked for * may be allowed for a named bot (or vice versa), and that distinction decides whether a fetch is a violation.

Pay attention to the AI-relevant agents you actually name, common ones include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, and Google-Extended. If your file only has a global Disallow, every one of those bots is measured against the same rule. Keep this list; it is the ruler for Step 3.

Step 2: Pull AI-bot requests from your logs

Get every AI-crawler request over a fixed window (28 days is a good default) from your server access logs or CDN bot analytics. For each request you need three fields: the user-agent, the path requested, and the HTTP status returned. Group the requests by operator so one bot's behaviour doesn't hide in the total, the same user-agent grouping we detail in seeing which AI bots crawl your site.

Keep the raw paths, not just counts. You need to know which URLs each bot hit to compare them against your disallow list, so a summary of "GPTBot: 4,000 hits" isn't enough on its own.

Step 3: Match each request against your disallow rules

Now cross-reference. For each AI-bot request, ask: was this path disallowed for this user-agent, and did it return content? A non-compliance event is the overlap:

non-compliant hit = HTTP 200 response to a path your robots.txt disallowed for that bot

Tally those hits per operator and divide by that operator's total requests to get its non-compliance rate for your site. A bot that fetched only allowed paths complied, however heavily it crawled. A bot that pulled disallowed paths and got a 200 ignored your file. Compare your per-bot rates to the ~30% network aggregate: far lower means your rules are largely respected; near or above it means real leakage worth acting on.

The signal is narrow and unambiguous: a 200 status on a path you disallowed for that agent. Everything else, heavy crawling of allowed pages, blocked paths that returned 403, is not a violation.

Step 4: Verify identity before you count it

Don't trust the user-agent string. Because a self-declared name is trivially spoofed and non-compliant scrapers rotate IPs and fake headers, confirm each "bot" against the operator's published IP ranges with a reverse-DNS check before you attribute a violation to it. Sort the results into two buckets:

Verified bot, hit a disallowed path — genuine non-compliance by a named operator. This is what the network data measures.
Unverifiable IP claiming a bot name — likely impersonation or a scraping vendor wearing a costume, a different threat you handle with edge blocking, not robots.txt tuning.

Separating the two keeps you from blaming an operator for traffic that wasn't theirs, and tells you whether your problem is real bots bending the rules or imposters ignoring them entirely.

Step 5: Decide, enforce, and re-measure

A non-compliance number is only useful if it drives a decision, path by path. For content you genuinely need closed (proprietary data, paywalled pages), move enforcement to where it works: a CDN or WAF rule keyed to verified identity that returns a 403, since robots.txt can't enforce anything and a bot ignoring it won't stop on request. For everything else, remember that the crawlers reading you are usually the ones that ground and cite live answers, so an over-eager block quietly costs you citations, and check you aren't already blocking crawlers by accident.

Then recompute each month against the same window. Non-compliance is a moving target that rose ten-fold across 2025, so a single snapshot is a starting point, not a verdict. Watching who crawls you, whether they respect your rules, and whether that reading turns into citations, across engines and over time, is the ongoing job that Buffy Intel is built for. Questions: [email protected].