The public crawl-to-refer ratios are network-wide averages, and your site can sit far from them. Computing your own takes two data sources you already have, server or CDN logs for the crawl side and analytics for the referral side, divided over the same window. This is the hands-on companion to how many pages each AI bot crawls per visitor; read that for the benchmark figures. Below is the method, in five steps.
Step 1: Count crawler hits per operator from your logs
Pull every request from verified AI crawler user-agents over a fixed window (use 28 days to match the public data). Your server access logs or CDN bot analytics record each one. Group the hits by operator, not just by bot name, since one operator often has several tokens:
| Operator | User-agent tokens to sum |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User |
| Perplexity | PerplexityBot, Perplexity-User |
| Google (AI) | Google-Extended (training) + AI-Mode/AIO fetches |
| Microsoft | BingBot (shared with Copilot) |
Count only HTTP 200 responses, so a block doesn't inflate the picture, and confirm the requests really come from the operator's published IP ranges rather than a spoofed string. Our guide to seeing which AI bots crawl your site covers the filtering and IP-verification mechanics in detail.
Step 2: Count AI referrals per operator from analytics
For the same dates, count the sessions each operator's assistant sent you. Web-based engines usually arrive with a referrer (perplexity.ai, chatgpt.com, Google AI surfaces), so a referrer filter or a dedicated AI channel captures them. A large share of AI visits carry no referrer and fall into "Direct," so add an estimate for that dark slice rather than ignoring it. The fastest way to stand this up is the GA4 AI-referral dashboard, which builds both the visible channel and a dark-traffic estimate.
Keep the operators separate. You want ClaudeBot's crawls matched to Anthropic's referrals, not one blended total, because the ratios differ by an order of magnitude between operators.
Step 3: Divide, per operator
For each operator, divide total crawler hits by total referral sessions over the window:
crawl-to-refer ratio = crawler HTTP-200 hits ÷ AI referral sessions
A result of 900:1 means the operator fetched about 900 of your pages for each visitor it sent. Do this per operator and you get your own version of the public table, tuned to your site. If an operator sent zero referrals but crawled heavily, note the ratio as "crawl-only" rather than dividing by zero; that is common and expected for training-first bots.
Match the crawl side and the referral side to the exact same 28-day window, and always split by operator: a blended ratio hides the ten-fold gap between a training-first bot like ClaudeBot and a search-backed one like Google.
Step 4: Benchmark against the public figures
Compare your per-operator ratios to the mid-2026 public numbers (Cloudflare Radar, window ending 21 July 2026): roughly 2,237:1 for Anthropic, 217:1 for OpenAI's GPTBot, 225:1 for Perplexity, 35:1 for Microsoft, and 4.6:1 for Google. What the comparison tells you:
- Much higher than the benchmark usually means strong crawling but weak referral capture, often a dark-traffic measurement gap, not a visibility problem. Fix the measurement before drawing conclusions.
- Much lower than the benchmark can mean the crawler isn't reaching you at all (check for robots.txt or CDN blocks), so your denominator is small because you're barely crawled.
- In line with the benchmark is the common, healthy case: you're being learned at the normal rate.
Step 5: Track the trend, not the snapshot
Recompute monthly against the same 28-day window and watch the direction. The public ratios fell sharply across 2026 as AI assistants matured into click-sending search products; your own ratio should drift down too as engines send you more referrals. A steadily falling ratio is good news. A ratio that spikes is usually a training pass, harmless, while a crawl side that suddenly drops to near zero is worth investigating as a possible block.
Do not treat any single month as a verdict. The number is context for two separate questions, can AI reach me, and does AI cite me, and only the second is the real goal. Heavy crawling with no citations is a leading indicator, not a failure; the point of tracking the ratio is to confirm the crawl is happening while you measure the citations that actually matter.
Computing the ratio once is a useful audit; watching it, your citations, and your AI referrals move together across engines is the ongoing job that Buffy Intel is built for. Questions: [email protected].