A third option is emerging between letting AI crawlers in free and blocking them: charging them. As of mid-2026, Cloudflare's Monetization Gateway (in waitlist) lets a site quote a price for a page, dataset, API, or MCP tool and settle it in stablecoins over the open x402 protocol, which revives the HTTP 402 Payment Required status. A crawler requests a resource, receives a 402 with the price, pays, and retries with proof attached. It is metered, per-request access — a middle posture that classic robots.txt blocking never offered.
This piece explains how pay-per-crawl works, who is building it, and how to weigh charging for access against wanting to be cited. The products named are early and gated, so read this as the shape of an emerging option, dated to mid-2026, not a settled control.
What is pay-per-crawl?
Pay-per-crawl charges an AI crawler for each fetch, instead of allowing or denying it for free. Today most sites face a binary: robots.txt or a CDN rule either lets a bot in (you earn nothing) or shuts it out (you get nothing and forfeit citations). Pay-per-crawl inserts a third state — the bot can reach the content, but only after paying a price you set, settled automatically inside the request.
The mechanism is x402, the HTTP-native payment standard the Linux Foundation began stewarding in July 2026. When a paid resource is requested, the server answers 402 Payment Required with a small payload naming the price, the accepted asset, and where to pay. The client pays — typically a stablecoin such as USDC, clearing sub-second — and retries with proof. No login, no checkout page, no invoice.
Who is building pay-per-crawl?
Cloudflare is furthest along, and it is not alone. Cloudflare first shipped Pay Per Crawl in 2025 as a way to charge AI bots for content, alongside a July-2025 shift to blocking AI crawlers by default for new domains. In mid-2026 it generalised that into the Monetization Gateway: in its own words, "instead of only charging crawlers for content, you will be able to charge any caller for any resource" — pages, datasets, APIs, or MCP tools — settled in stablecoins over x402.
| What | Detail (mid-2026) |
|---|---|
| Product | Cloudflare Monetization Gateway |
| Status | Waitlist; no general-availability date disclosed |
| Charges for | Web pages, datasets, APIs, MCP tools behind Cloudflare |
| Protocol | x402 (HTTP 402; open standard under the Linux Foundation) |
| Settlement | Stablecoins (USDC), sub-second |
| Predecessor | Cloudflare Pay Per Crawl (2025) |
Source: Cloudflare's Monetization Gateway announcement and x402 documentation, mid-2026 (waitlist-stage; details subject to change). Cloudflare and AWS both moved to embed x402 micropayments at the network edge within roughly two weeks of each other, per InfoQ reporting in July 2026, which is why edge-level metering — not a per-site plugin — is the likely shape of this.
Why would a site charge crawlers at all?
Because AI crawlers take far more than they give back. Cloudflare notes that AI crawlers "already request content anywhere from a hundred to tens of thousands of times for every visitor they send back" — the extraction-heavy pattern our crawl-to-refer ratio data quantifies, with pure-AI bots crawling thousands of pages per referral. When reading rarely turns into a visit, some publishers would rather be paid for the reading.
Metering also gives leverage that robots.txt lacks. A disallow rule is only a request, which a rising share of crawlers ignore; a price enforced at the edge is a wall with a turnstile. For owners of genuinely valuable data — archives, proprietary datasets, paid APIs — charging is a way to turn heavy crawling into revenue instead of cost.
Should you charge AI agents, or stay open for citations?
It depends on whether a given page's value is as data or as visibility — and for most marketing content, it's visibility. A price gate that a crawler won't pay removes you from the free AI answer entirely, and being cited and recommended in that answer is usually worth more than a micropayment. Charge the wrong pages and you buy pennies while losing the discovery that drives the business.
Pay-per-crawl is powerful for content that is a product, and self-defeating for content that is a billboard. The skill is knowing which of your pages is which.
A workable split for most sites:
- Keep open — educational, marketing, and product pages whose job is to be discovered, cited, and recommended by AI. Citations live in the free tier.
- Consider metering — proprietary datasets, research archives, and APIs that are genuinely a product, and that AI operators crawl heavily without referring traffic back.
- Measure either way — track whether metered resources still get requested (demand at your price) and whether open pages still get cited (visibility preserved).
The deeper point is that charging for access and earning citations are two different games, and most brands are still playing only for citations. Whether AI agents and answer engines actually surface, cite, and recommend the pages you keep open is exactly what Buffy Intel measures, snapshot over snapshot. Questions: [email protected].