Field note

License, litigate, or block AI crawlers: which posture fits your site?

Content owners are taking three distinct postures toward AI crawlers in 2026: license access for a fee, litigate over unauthorized scraping, or block at the CDN. This is a neutral, criteria-based comparison of what each posture costs, what it protects, what it gives up in AI citations, and who each one actually fits.

Buffy Editorial2026-08-13 · 4 min read

Content owners are taking three distinct postures toward AI crawlers in 2026 — license, litigate, or block — and they optimise for different things. Licensing trades access for revenue; litigation contests access that was allegedly taken covertly; blocking withholds access to protect content. None is universally right: the posture that fits depends on whether your value is the content itself, your legal resources, and how much you want to be cited in AI answers. This is a neutral, criteria-based comparison, not a recommendation of one path.

Last reviewed: 13 August 2026. The postures below reference public 2026 developments (licensing deals, the News Corp and Reddit lawsuits, Cloudflare's crawler-charging models); terms and defaults are moving, so treat specifics as dated. Each posture links to the deeper corpus piece on it.

What are the three postures, side by side?

Each posture answers a different question: how do I get paid, protect my work, or stay visible? Here is the trade-off on the criteria that matter.

Criterion License Litigate Block
Core goal Monetise access Contest covert access Protect content
Effect on AI citations Preserved / encouraged Neutral Reduced (you become unreadable)
Who it fits Sites with in-demand, licensable content Large owners with evidence + resources Content-is-the-product sites
Cost / effort Deal-making or metering setup High (legal) Low technically, high in lost visibility
Enforcement needed Metering / verified identity Verified-identity evidence Verified-identity CDN rules
2026 example News Corp–OpenAI, Google–Reddit News Corp v Brave; Reddit v Perplexity CDN default-block on ad pages

The one-line summary: licensing and litigating are both ways to get value from access, while blocking gives up access — and citations — to keep content in. All three depend on the same technical foundation: knowing who is really fetching you.

When does licensing access make sense?

Licensing fits when your content is in demand and identifiable enough to charge for, and you would rather be paid for access than withhold it. In 2026 the direct-deal version is largely a large-publisher move: reported deals include News Corp–OpenAI (over $250M across five years) and Amazon–The New York Times ($20–25M/year), with most terms undisclosed.

For everyone else, the equivalent is metered access rather than a bespoke contract. Cloudflare's pay-per-crawl and emerging pay-per-answer models let a site price or charge for crawler access at the edge, using the HTTP 402 Payment Required status and the x402 settlement standard. Licensing keeps you readable — and therefore citable — while capturing value, which is why it is the posture most compatible with wanting AI visibility. It is also the least mature: gated, early, and priced differently by provider.

When is litigating the right posture?

Litigation fits large content owners with clear evidence that access controls were bypassed, not most sites. The 2026 cases show the shape: News Corp sued Brave (filed 22 July 2026) alleging disguised crawlers evaded its blocks, and a federal judge on 31 July 2026 largely refused to dismiss Reddit's "industrial-scale" scraping suit against Perplexity and its data-scraping suppliers. The full detail is in the 2026 AI-scraping lawsuits.

For a smaller site, litigation is rarely practical. The realistic equivalents are technical enforcement and documentation: confirm crawlers by verified identity and record non-compliance in your logs, per checking whether AI bots ignore your robots.txt. The indirect benefit of the big cases is real, though: if courts treat evading a block as unlawful, covert scraping gets riskier for everyone, strengthening sites that simply enforce clear rules.

When should you block — and what does it cost?

Blocking fits sites where the content itself is the product — paywalled archives, proprietary datasets — more than brands that want to be recommended. The cost is direct: a crawler you block can't read you, so it can't cite you. Blocking AI crawlers trades AI visibility for content protection.

Two caveats make blocking harder than it looks:

  • A blanket block is rarely clean. A single user-agent can span search indexing, training, and agent fetching, so blocking "the AI bot" can also cut the search crawlers that feed AI Overviews. Decide per crawler class.
  • Enforcement needs verified identity. Because a self-declared name is spoofable, a real block requires verified-bot rules at the CDN (reverse-DNS-confirmed IP ranges), not a robots.txt line that non-compliant scrapers ignore. See is your CDN blocking AI crawlers for the failure mode where this happens by accident.

Can you combine postures?

Yes, and most sophisticated owners do. The common pattern in 2026 is license to the operators you want, enforce verified-identity access against the rest, and reserve litigation for clear, evidenced bypassing — News Corp licenses to OpenAI and Meta while suing Brave. The postures are not mutually exclusive; they are tools applied per crawler and per content type. The one constant across all three is that they only work if you can tell who is really fetching you.

Buffy Intel measures the visibility side of whichever posture you pick — whether your content is being reached and cited across AI engines, so you can see what a block costs you or what licensed access earns you. Start with a snapshot of where you appear today. Questions: [email protected].

Frequently asked

Should I block AI crawlers to protect my content?

Only after weighing what you give up. Blocking at the CDN stops well-behaved crawlers from reading your pages, which also stops them from citing you in AI answers, so it protects content at the cost of AI visibility. It fits sites whose value is the content itself (paywalled archives, proprietary data) more than brands that want to be recommended. It is also incomplete on its own: a self-declared user-agent is spoofable, so enforcement needs verified-identity rules, and non-compliant scrapers may ignore the block entirely. For most brands seeking citations, selective blocking by crawler class beats a blanket block.

Can a small site actually license its content to AI companies?

Directly negotiating a deal is largely a big-publisher move today; the reported 2026 licensing deals involve large media companies and dollar figures in the millions. But the infrastructure for smaller sites to charge for access is emerging: Cloudflare's pay-per-crawl and pay-per-answer models let a site meter or price crawler access without a bespoke contract. Treat direct licensing as a large-publisher lever for now, and metered access as the option to watch for everyone else. Both are early and their terms are moving, so hedge any plan against change.

Is litigating over AI scraping realistic for anyone but large publishers?

Rarely as a first move. The 2026 cases (News Corp v Brave; Reddit v Perplexity) are brought by large content owners with legal resources and clear evidence of bypassed access controls. For most sites, litigation is not a practical posture; the realistic equivalents are enforcing access technically and documenting non-compliance in your own logs. The value of the big cases for smaller sites is indirect: if courts treat evading a block as unlawful, covert scraping gets riskier for everyone, which strengthens the position of sites that simply enforce clear rules.