How do I stop my CDN from blocking AI crawlers I want?

Set explicit bot rules instead of inheriting the default. In Cloudflare, that means reviewing the managed AI-bot controls and any bot-management or WAF rules, then explicitly allowing the live-retrieval and agent crawlers you want on the pages you want them on, rather than leaving the September 2026 defaults in place. Then verify with your own logs that those user-agents are actually getting 200 responses, not a challenge or a block, on your key pages.

Which AI crawlers should I keep allowed for AI visibility?

At minimum the live-retrieval and answer bots that ground and cite live responses, such as OAI-SearchBot and ChatGPT-User (OpenAI) and PerplexityBot (Perplexity), plus classic search crawlers like Googlebot and Bingbot that feed AI Overviews and AI Mode. Training crawlers such as GPTBot and Google-Extended are a separate call: allowing them helps future models learn about you, blocking them protects proprietary content. Decide per class, not with one switch.

Do I need to do anything if I'm an existing paid Cloudflare customer?

Cloudflare's new defaults target new customers, new sites, and free users who haven't changed settings, so an existing paid site with its own bot rules isn't converted automatically. But it's still worth auditing: defaults drift, sites get re-provisioned, and a page you assume is open to citations may already be behind a challenge. A five-minute log check per key page beats discovering months later that agents can't reach you.

How to keep AI crawlers reaching your site after the September 2026 default-block

If you want AI citations, don't let a changed default decide for you. Cloudflare's 15 September 2026 defaults block AI training and agent crawlers on ad-bearing pages for new and free-tier sites, while leaving search crawlers allowed. This is a five-step check to make sure the crawlers you actually want can still reach and cite your pages, whatever your CDN's default becomes.

This assumes you want AI visibility on most of your content. If your goal is to keep content out of AI entirely, the steps still apply — you just decide the other way at step 3.

Step 1: Confirm whether the new default reaches you

Start by checking if you're in the affected subset. The September defaults apply to new Cloudflare customers, new sites added by existing customers, and free-tier users who haven't changed their settings — not to existing paid sites that already set their own bot rules. If you're a new or free site on Cloudflare, assume the default now leans toward blocking AI training and agent bots on ad pages, and plan to override it deliberately. If you're an established paid site, you're likely unaffected automatically, but audit anyway (defaults drift, and sites get re-provisioned).

Step 2: Map which of your pages carry ads

The default only bites on ad-bearing pages, so find them. List the templates or sections of your site that serve advertising — often article pages, free tools, or high-traffic guides. These are the pages where training and agent crawlers will be blocked by default, and often the very pages you most want cited in AI answers. Pages with no ads aren't touched by this particular default, so you can prioritise the audit on the ad-supported set.

Step 3: Decide per crawler class, not with one switch

Blocking is not one decision — it's several. Sort the bots by what they're for and choose per class:

Crawler class	Keep allowed if you want…	Examples
Live-retrieval / answer	Citations in live AI answers	OAI-SearchBot, ChatGPT-User, PerplexityBot
Search indexing	AI Overviews / AI Mode presence	Googlebot, Bingbot
AI agent	Agents to complete tasks on your pages	Assorted agent user-agents
AI training	Future models to learn about you	GPTBot, Google-Extended, ClaudeBot

For a brand chasing visibility, the live-retrieval and search rows are usually a clear allow. Training crawlers are the genuine judgment call — allow them to be learned, block them to protect proprietary work. Use the full AI crawler user-agent directory to get the tokens right, and read the trade-offs in should you let AI crawlers index your site.

Step 4: Set explicit rules so you don't inherit the default

Once you've decided, write it down in your controls rather than leaving the default in place. In Cloudflare that means the managed AI-bot settings plus any bot-management or WAF rules; in robots.txt it means per-user-agent groups. The point is the same: an explicit allow for the crawlers you want, on the pages you want, so a future default change can't silently close them. Remember that a robots.txt line is only a request — the CDN-level rule is what actually enforces on ad pages.

A default is a decision someone else made for you. On the pages you monetise and want cited, replace it with one you made on purpose.

Step 5: Verify what's actually allowed

Don't trust the settings screen — check the logs. Fetch a few key pages as the crawler user-agents you care about, or read your server/CDN logs, and confirm those bots get a 200, not a challenge, a 403, or a silent block. This is the same discipline as catching an accidental CDN block: the only proof that a crawler can reach you is a successful fetch in the record. Re-check after 15 September 2026, and again whenever you re-provision or change plans. For an ongoing view of which bots actually reach you, see how to see which AI bots crawl your site.

The short version

Confirm you're affected, find your ad pages, decide per crawler class, set explicit rules, and verify with logs. Do that once and a shifting default can't quietly cost you citations. Whether AI engines and agents keep reaching, citing, and recommending your pages after infrastructure defaults change underneath you is exactly what Buffy Intel monitors, snapshot over snapshot. Questions: [email protected].