If my robots.txt allows AI bots, why would they still be blocked?

Because robots.txt is read by the crawler *after* it connects. A CDN or WAF can refuse the connection first. Returning HTTP 403 based on the user-agent. Before robots.txt is ever fetched. The two controls live at different layers, and the edge wins.

Does blocking AI bots at the CDN hurt my Google ranking?

Usually not directly. These rules typically still allow Googlebot. But they block the AI engines (ChatGPT, Claude, Perplexity) from reading and citing you, so you quietly lose AI-answer visibility while classic search looks fine. That split is exactly why it goes unnoticed.

How do I check which AI crawlers my site allows?

Send a request with each crawler's user-agent and read the status code: a browser-like UA and Googlebot should return 200; if GPTBot, ClaudeBot, or PerplexityBot return 403 (often with a 'server: cloudflare' header), your edge is blocking them.

Your CDN might be blocking the AI crawlers you want

You did everything right. You opened up robots.txt, added an llms.txt, marked up your pages with structured data. And the AI engines still don't seem to see you.

Here's the trap almost nobody checks: the decision about whether a crawler even reaches your site is often made one layer above your site. At your CDN.

robots.txt is a request; your CDN is the bouncer

robots.txt is a polite instruction a well-behaved crawler reads after it connects to your server. But a CDN or web application firewall (Cloudflare and friends) decides whether that connection is allowed at all: before your server, or your robots.txt, ever enters the picture.

So if the CDN returns a 403 Forbidden to a crawler's user-agent, your wide-open robots.txt is irrelevant. The bot never gets far enough to read it. You can have the most welcoming robots file on the internet and still be a locked door.

Why this is suddenly everywhere

In the last year, CDNs shipped one-click "block AI bots" controls, and some made blocking the default. As of July 1, 2025, Cloudflare began blocking AI crawlers by default for new domains: at sign-up you're explicitly asked whether to allow them, and the default posture is deny. (Existing domains weren't auto-flipped, but were handed the same controls.) Cloudflare even lets you decide separately by what the bot does: training vs. fine-tuning vs. inference, and offers a "pay-per-crawl" option to charge for access instead of a flat allow/deny. Add the wave of security plugins doing similar, and a lot of sites are now blocking AI bots without anyone on the marketing side knowing.

The tell-tale signature is specific:

Browsers get through (200).
Googlebot gets through (200). These rules usually whitelist classic search.
GPTBot, ClaudeBot, PerplexityBot get a 403.

Perfect for blocking scrapers. Catastrophic if you actually want to show up in AI answers.

The 60-second check (do this now)

Pretend to be each AI crawler and see what your site returns:

curl -sI -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yourdomain.com | head -1
curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://yourdomain.com | head -1
curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0)" https://yourdomain.com | head -1

How to read it:

HTTP/2 200 → the crawler can reach you. Good.
HTTP/2 403 → you're blocking it, regardless of what robots.txt says. If you also see a server: cloudflare header on that 403, your CDN is the one saying no.

Now compare against a normal browser user-agent. If the browser returns 200 but the AI bots return 403, your edge is the culprit: not your code, not your robots file.

How to fix it

Cloudflare: Security → Bots → turn off Block AI bots / AI Scrapers and Crawlers (or set it to Allow). To keep general bot protection, add a WAF rule that explicitly allows the user-agents you care about: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, CCBot.
Other CDNs / WAFs / security plugins: find the "AI bots" or "known bots" rule and allow-list those same agents.
Decide per bot, not all-or-nothing. You might welcome live-retrieval and citation bots while making a separate call on training crawlers, that trade-off is worth a deliberate decision (more in Should you let AI crawlers index your site?).
Re-test with the same curl commands. You're looking for 200s.

The bigger point

Allowing access is necessary but not sufficient. A bot that can reach a page still has to be able to parse it. But access comes first. You can optimize every page on your site, and if the front door is returning 403 to the AI engines, none of it lands.

An open robots.txt that's blocked at the CDN is the most common "invisible to AI" failure there is, and the easiest to miss, because it's silent and it lives in a dashboard marketers never open.

Check the door before you decorate the house. It's exactly the kind of silent gap an AI-readiness audit is built to catch, and the kind of regression ongoing monitoring with Buffy Intel flags the moment it reappears.