As of mid-2026, AI crawlers make up about 22% of all bot traffic. Googlebot still leads the crawler leaderboard; Anthropic's ClaudeBot holds the #2 spot among purpose-built AI crawlers after its June surge cooled, and Anthropic's combined footprint (ClaudeBot + Claude-SearchBot, ~19.8%) now trails only Googlebot. This is a dated, fully-attributed reference to which bots are hitting sites, what each one is for, and what the shift means for your access decisions.
Last reviewed: 15 August 2026. Share figures are for the full month of July 2026 from Cloudflare Radar's AI Insights (per-crawler share of AI-bot requests, pulled 1 August 2026), plus Cloudflare's own network analysis. Cloudflare sees a large share of web traffic but not all of it, so treat these as a single-network snapshot. Directional, and shifting month to month.
Which AI crawlers have the most traffic right now?
Here is the July 2026 share of AI-bot traffic, with the month-over-month change from June:
| Crawler | Operator | July 2026 share | Change vs June | Primary job |
|---|---|---|---|---|
| Googlebot | ~24.5% | −0.4 pp | Index (Search + AI Overviews / AI Mode) | |
| ClaudeBot | Anthropic | ~16.3% | −3.7 pp | Train |
| Meta-ExternalAgent | Meta | ~12.7% | +2.5 pp | Train |
| GPTBot | OpenAI | ~9.7% | +0.1 pp | Train |
| Bingbot | Microsoft | ~8.9% | +0.8 pp | Index (Bing + Copilot grounding) |
| Applebot | Apple | ~6.5% | +0.7 pp | Index / train |
| Amazonbot | Amazon | ~6.0% | +0.5 pp | Index / train |
| Bytespider | ByteDance | ~5.1% | −2.2 pp | Train |
| Claude-SearchBot | Anthropic | ~3.5% | +0.2 pp | Index (Claude citations) |
Source: Cloudflare Radar AI Insights, July 2026 (full-month share of AI-bot requests). The one-line read: Googlebot still leads because it feeds both classic Search and Google's AI features, but among purpose-built AI crawlers, ClaudeBot is still the front-runner — and Anthropic's two bots together (~19.8%) are now the clear #2 crawl presence on the web.
What just changed, and why does Anthropic's footprint matter?
The story is still Anthropic, now told in two bots instead of one. ClaudeBot's June spike partly normalised — it eased from about 20.0% to 16.3% of AI-bot traffic in July — but that does not undo the shift. Add Claude-SearchBot (~3.5%) and Anthropic's combined crawl footprint is about 19.8%, second only to Googlebot and comfortably ahead of OpenAI's GPTBot. Meta-ExternalAgent climbed to roughly 12.7% to take clear third, while Bytespider kept fading.
That footprint lines up with Anthropic's rising business AI adoption over the same period. More usage tends to mean more aggressive crawling to keep models and retrieval fresh. For your site, the practical signal is that Anthropic is now fetching your pages heavily across two bots, so whether Claude can learn from and cite you increasingly depends on your letting both ClaudeBot (train) and Claude-SearchBot (index) through.
Googlebot still tops the list, but the durable story is Anthropic's two-bot footprint (~19.8%) settling in as the clear #2 crawl presence — even after ClaudeBot's one-month spike cooled. Crawl attention is following adoption.
How big has AI crawling become overall?
Large, and increasingly training-driven. Cloudflare reports that AI crawlers reached about 22% of all bot traffic in Q1 2026, up from a negligible share two years earlier. Now bigger than every bot category except traditional search-engine crawlers. And the mix has tilted toward training: roughly 80% of AI bot activity is training-purpose crawling, up from about 72% a year earlier.
The longer trend, year over year (Cloudflare, July 2024 to July 2025):
- GPTBot grew from ~4.7% to ~11.7% of overall crawl share.
- Meta-ExternalAgent jumped from ~0.9% to ~7.5%.
- Bytespider collapsed from ~14.1% to ~2.4% as ByteDance pulled back.
The takeaway: the set of bots visiting your site turns over fast, and a crawler that dominated last year can fade while a new one takes over. This is why a static robots.txt written once and forgotten drifts out of date.
Does heavy crawling mean heavy traffic back?
No, but the gap is narrowing fast. Cloudflare's crawl-to-referral data shows AI operators still crawl far more than they send visitors — yet the ratios improved sharply between June and July 2026 as AI search began sending more referrals:
| Operator | Crawl-to-refer, June 2026 | Crawl-to-refer, July 2026 |
|---|---|---|
| Anthropic | ~3,386 : 1 | ~1,917 : 1 |
| OpenAI | ~647 : 1 | ~251 : 1 |
Source: Cloudflare Radar crawl-to-refer summary, pulled 1 August 2026. Both operators roughly halved their pages-crawled-per-referral in a single month, so the "crawl a lot, refer almost nothing" pattern is easing — though Anthropic still crawls on the order of thousands of pages per referral. High crawl volume remains a leading indicator of being learned and indexed, not a promise of clicks, as we cover in where AI crawls versus where it sends visitors. Judge a bot's crawling by whether it helps you get cited, not by referral traffic alone.
How should you use this data?
Treat it as a dated snapshot that informs access decisions, not a target:
- Decide per bot, not with one switch. Each crawler has a job. Train, index, or answer. Allow at least the index and answer bots that drive citations; the AI crawler user-agent directory and the train/index/answer lifecycle explain what each costs you if blocked.
- Don't let a CDN silently block the surging bots. A default "block all unknown bots" rule can cut off exactly the fast-growing crawlers like ClaudeBot and Claude-SearchBot. See is your CDN blocking AI crawlers.
- Watch your own logs, not just the industry chart. Which bots hit your site, and how often, is the signal that matters. The method in how to see which AI bots crawl your site.
- Revisit quarterly. Because shares move this fast, a crawl policy set once decays; review it as the leaderboard shifts.
The industry share chart tells you where crawler attention is heading in aggregate. What decides your AI visibility is which bots reach your pages and whether that access turns into citations for your brand. Measured on your own site and across engines over time. That is exactly what Buffy Intel is built to track.