AI crawling is not being switched off across the web, but its default is changing. From 15 September 2026, Cloudflare will start blocking AI training and AI-agent crawlers by default on any page that carries advertising, while leaving classic search crawlers allowed. The change applies to new customers, new sites from existing customers, and free-tier users who have not set their own rules. Alongside it, Cloudflare is shifting how it charges AI companies, from pay-per-crawl to pay-per-answer.
This piece explains what actually changes on that date, who it reaches, and what a brand that wants to be cited in AI answers should watch for. The facts are attributed to Cloudflare's July 2026 announcements and corroborating reporting, and dated to mid-2026, because default policies and billing models are still moving.
What is Cloudflare changing on 15 September 2026?
Cloudflare is flipping the default posture for AI crawlers on ad-supported pages. Under the new defaults, a search crawler that indexes your pages is still welcome, while a crawler fetching content to train a model or to power an AI agent is blocked unless the site owner opts in.
| Crawler class | Default on ad pages (from 15 Sep 2026) | Example jobs |
|---|---|---|
| Search indexing | Allowed | Classic search results; feeds AI Overviews / AI Mode |
| AI training | Blocked | Gathering content to train future models |
| AI agent | Blocked | Fetching pages to complete a user's task live |
| Mixed-use (blends the three) | Blocked unless the owner opts in and the bot lets users separate the functions | One user-agent doing several jobs |
Who it applies to, in Cloudflare's own framing: new customers, new sites added by existing customers, and every free user who has not changed their settings. Existing paid sites with their own bot rules are not converted automatically. Source: Cloudflare's 1 July 2026 announcement and reporting by TechCrunch and PPC Land (single-operator policy; details may change before the date).
What is a "mixed-use" crawler, and why does it matter?
A mixed-use crawler is one that blends several jobs — search indexing, model training, and AI-agent fetching — behind a single user-agent. It matters because Cloudflare's new default keys off purpose: search is allowed, training and agent use are not. A bot that does all three at once cannot be cleanly allowed for one job without allowing the others, so under the September defaults it is blocked on ad pages unless the operator lets site owners separate the functions.
This is the same knot the corpus has flagged before: a single token like Googlebot can span indexing, training-adjacent, and AI-answer use, which is why "just block the AI bots" is rarely clean. The September change turns that ambiguity into a default: when a crawler's purpose is unclear or mixed, the safe default becomes deny on the pages a publisher monetises.
What is "pay per use," and how is it different from pay per crawl?
Pay-per-use charges an AI company when its content is actually used to answer a question, not each time a crawler fetches a page. Cloudflare described this shift on 1 July 2026, evolving its earlier pay-per-crawl model, which billed per fetch using the HTTP 402 Payment Required status.
The rationale is a waste statistic: Cloudflare says more than half of the crawl traffic from bots it classifies as legitimate re-fetches pages that have not changed since the last visit. Charging per crawl therefore prices re-fetching, not value. Pay-per-use ties payment to the answer the content helped produce. Early partners named for this model include Ceramic.ai (publishers paid when their content appears in its search results) and You.com (agents buy premium content on demand).
| Model | What triggers payment | Status (mid-2026) |
|---|---|---|
| Pay Per Crawl (2025) | Each crawler fetch (HTTP 402) |
Live, being superseded |
| Pay Per Use / pay-per-answer | Content used to answer a query | Announced Jul 2026; early partners |
Source: Cloudflare, July 2026, corroborated across TechCrunch and PPC Land. This sits on top of the same settlement plumbing — the x402 payment standard and Cloudflare's Monetization Gateway — covered in our pay-per-crawl explainer; pay-per-use is the pricing evolution, not a different pipe.
Why single out ad pages?
Because an ad is treated as a signal that the page was built for a human to land on. Cloudflare's reasoning is that content earning ad revenue is where the crawl-to-refer imbalance bites hardest: the same crawl often feeds a model that then answers the user directly, so the visit — and the ad revenue — never arrives. Making ad pages opt-in for training and agent bots is meant to protect that revenue while keeping search discovery intact.
That imbalance is well documented. Pure-AI crawlers fetch thousands of pages for every visitor they refer, and a robots.txt disallow is only a request that a rising share of bots ignore. A CDN-level default enforced at the edge is a harder wall than a text file — which is exactly why the default flipping matters more than a robots.txt line.
Does this block you out of AI answers?
Mostly not — but it can, quietly, on your monetised pages. The default only touches ad-bearing pages, only under the new-defaults subset, and it leaves search crawlers allowed, so the pipeline into Google's AI Overviews and AI Mode is unaffected. The risk is narrower and easier to miss: if your ad-supported pages are silently closed to live-retrieval and agent bots, you can forfeit citations and agent visibility on precisely those pages, the same failure mode as an accidental CDN block, now arriving as a default rather than a mistake.
The default is flipping from "let AI read everything" to "let search in, make AI training and agents ask." For a brand that wants citations, the danger isn't the block itself — it's inheriting a default you never chose.
The decision is still yours to make per crawler class, exactly as our guide to allowing vs. blocking AI crawlers lays out. What has changed is that doing nothing no longer means "open by default" everywhere.
What should brands do about it?
Two moves. First, find out whether you are in the affected subset and what your CDN is actually allowing after the date — see our step-by-step on keeping AI crawlers reaching your site. Second, decide deliberately: keep live-retrieval and agent bots welcome on the pages whose job is to be discovered and cited, and reserve blocking or metering for content that is genuinely a product.
Whether AI engines and agents can still reach, cite, and recommend your pages after a default like this changes underneath you is exactly what Buffy Intel measures, snapshot over snapshot. Questions: [email protected].