AI crawlers take far more than they give back, and the exact gap is now measurable per operator. Over the 28-day window ending 21 July 2026, Cloudflare Radar data shows Anthropic's ClaudeBot crawled roughly 2,237 pages for every visitor it referred, OpenAI's GPTBot about 217, and Google about 4.6. That number, the crawl-to-refer ratio, is the cleanest single measure of how differently AI operators and classic search engines treat your site.
This is a dated reference, not a new dataset. Last reviewed: 28 July 2026. The headline figures come from Cloudflare Radar's bot-and-crawler analytics for the 28-day window ending 21 July 2026, as compiled by SEOmator, and are cross-checked against SEOmator's own panel of 500+ B2B sites (January–July 2026). Every figure is a verified-bot, network-wide average that varies by site type, so read the direction as firmer than any single decimal, and cite "Cloudflare Radar, window ending 21 July 2026" when you reuse one. It sits alongside our AI crawler share leaderboard and our where-AI-crawls-vs-where-it-sends-visitors explainer.
What is the crawl-to-refer ratio?
The crawl-to-refer ratio is the number of pages an operator's AI crawler fetches for every visitor its assistant refers back to a site. Divide total crawl requests by total referral sessions over the same window and you have it. A ratio of 217:1 means the bot fetched about 217 pages for each click it sent. It is a single number that captures a structural fact: AI operators crawl to learn, and only some of that learning ever turns into a visit.
The metric matters because crawl volume and referral traffic are separate outcomes that people routinely conflate. A crawler reaching your pages tells you that you are eligible to be learned, indexed, and eventually cited. It says nothing about whether the assistant will send anyone your way. The ratio makes that split legible in one figure.
How many pages does each AI bot crawl per referral?
Wildly different amounts, split cleanly between pure-AI operators and search-backed ones. The table below is the mid-2026 snapshot.
| AI operator (crawler) | Pages crawled per referral | Category |
|---|---|---|
| Mistral | ~3,389 : 1 | Pure-AI |
| Anthropic (ClaudeBot) | ~2,237 : 1 | Pure-AI |
| Perplexity | ~225 : 1 | AI search |
| OpenAI (GPTBot) | ~217 : 1 | Pure-AI / AI search |
| Microsoft (Copilot) | ~35 : 1 | Search-backed |
| Google (Gemini / AI Overviews) | ~4.6 : 1 | Search-backed |
| DuckDuckGo | ~2.5 : 1 | Search-backed |
Source: Cloudflare Radar bot analytics, 28-day window ending 21 July 2026, compiled by SEOmator (verified-bot, network-wide averages; single-source, directional). The pattern is the story: operators without a mature search engine attached (Mistral, Anthropic) crawl thousands of pages per click, while Google, which has always paired crawling with a click-sending search product, sits near parity. SEOmator's independent panel of 500+ B2B sites (January–July 2026) landed on nearly identical numbers, Anthropic 2,363:1, Microsoft 35:1, Google 4.0:1, which is why the ordering, if not the exact decimal, is trustworthy.
Is the crawl-to-refer gap narrowing?
Yes, and quickly for the biggest AI crawlers. The same Cloudflare-derived series shows the pure-AI ratios falling hard across 2026 as the assistants matured into search products that actually send clicks:
| Operator | Earlier 2026 ratio | Window ending 21 Jul 2026 |
|---|---|---|
| Anthropic (ClaudeBot) | ~23,951 : 1 (Q1 2026) | ~2,237 : 1 |
| OpenAI (GPTBot) | ~1,276 : 1 (Jan–Mar 2026) | ~217 : 1 |
| ~4.9 : 1 (Mar 2026) | ~4.6 : 1 |
Source: Cloudflare Radar via SEOmator, Q1 2026 vs the window ending 21 July 2026 (directional; the ratio is volatile month to month). ClaudeBot's roughly ten-fold drop is the sharpest move: it crawled on the order of tens of thousands of pages per referral in early 2026 and mid-2025, and now crawls thousands. This lines up with the older, coarser reading we cited last year, Anthropic on the order of tens of thousands to one and OpenAI around a thousand to one in mid-2025, so the multi-quarter trend is a steady climb toward parity. Google, already near parity, barely moved. The gap is closing from the top down as AI search sends more traffic, not because the bots crawl less.
Why is the ratio so high in the first place?
Because AI operators crawl for two jobs that mostly happen before any click. The first is training: bots fetch huge swaths of the web to build the datasets a large language model learns from, and that pass touches far more pages than any user will ever visit. The second is retrieval: search-backed assistants maintain a fresh index to ground live answers, the retrieval-augmented generation layer, which also means fetching broadly and repeatedly.
Neither job is tied to sending you a visitor. A page can be crawled, learned, and summarised in an answer while the user never leaves the assistant, the Dark Library Effect we documented in the page-type data. So a high ratio is the normal physics of AI discovery, not a malfunction. It is also why heavy crawling with zero citations is a leading indicator, not a failure: the crawl reliably comes weeks before the citation.
The crawl-to-refer ratio is the price of being learned: an AI operator may fetch thousands of your pages before it sends a single visitor, and that is the expected shape of AI discovery, not a sign anything is broken.
What should you actually do with this number?
Treat it as context for two separate decisions, not a target to chase. Concretely:
- Don't judge an AI bot by referral clicks. A crawler sending few visitors is normal; the payoff shows up as citations and brand knowledge, so measure citations, not just clicks.
- Confirm the crawlers can actually reach you. A ratio only exists if the bots get HTTP 200s; check your logs so a silent robots.txt or CDN block isn't hiding you, using our guide to seeing which AI bots crawl your site.
- Watch bandwidth if the ratio is extreme. For a large site, a 2,000:1 crawler is real load; rate-limit or serve it efficiently rather than block it outright, or you lose the citations too.
- Remember referral counts understate reality. Much AI-referred traffic lands in analytics as direct traffic, so the true referral side of the ratio is higher than your dashboard shows; a GA4 AI-referral view narrows the gap.
- Compute your own rather than assuming the network average applies; site type moves the number a lot (see the companion how-to).
The deeper point is that "we get crawled constantly" and "AI recommends us" are different claims needing different evidence. Watching where AI crawls, what it cites, and where it sends visitors, across engines and over time, and separating crawl volume from real recommendation, is exactly what measuring AI visibility with Buffy Intel is built to do. Questions: [email protected].