As of mid-2026, AI crawlers make up about 22% of all bot traffic, and the leaderboard has just reordered: Anthropic's ClaudeBot surged to the #2 spot, overtaking Meta and OpenAI's GPTBot in the largest one-month share gain on record. This is a dated, fully-attributed reference to which bots are hitting sites, what each one is for, and what the shift means for your access decisions.
Last reviewed: 13 July 2026. All share figures are from Cloudflare Radar's AI Insights (a rolling 28-day window ending early July 2026) as compiled in public monthly crawler reports, plus Cloudflare's own network analysis. Cloudflare sees a large share of web traffic but not all of it, so treat these as a single-network snapshot. Directional, and shifting month to month.
Which AI crawlers have the most traffic right now?
Here is the June 2026 share of AI-related crawler traffic, with the month-over-month change:
| Crawler | Operator | June 2026 share | Change vs May | Primary job |
|---|---|---|---|---|
| Googlebot | ~24.9% | −1.8 pp | Index (Search + AI Overviews / AI Mode) | |
| ClaudeBot | Anthropic | ~20.0% | +8.0 pp | Train |
| Meta-ExternalAgent | Meta | ~10.2% | −2.4 pp | Train |
| GPTBot | OpenAI | ~9.6% | −0.5 pp | Train |
| Bingbot | Microsoft | ~8.1% | −0.1 pp | Index (Bing + Copilot grounding) |
| Bytespider | ByteDance | ~7.3% | −2.8 pp | Train |
| Applebot | Apple | ~5.8% | −1.0 pp | Index / train |
| Amazonbot | Amazon | ~5.5% | +0.2 pp | Index / train |
| Claude-SearchBot | Anthropic | ~3.3% | +0.6 pp | Index (Claude citations) |
Source: Cloudflare Radar AI Insights, June 2026 (rolling 28-day window). The one-line read: Googlebot still leads because it feeds both classic Search and Google's AI features, but among purpose-built AI crawlers, ClaudeBot is now the front-runner.
What just changed, and why does ClaudeBot's jump matter?
The story of the month is Anthropic. ClaudeBot rose from about 12.1% to 20.0% of AI crawler traffic in a single month: roughly a 66% relative gain, which Cloudflare's monthly report called the single largest one-month share gain any crawler has posted in the life of the report. Meta-ExternalAgent and GPTBot both eased back over the same window, and Googlebot slipped below 25% for the first time.
That surge does not stand alone. It lines up with Anthropic's rising business AI adoption over the same period. More usage tends to mean more aggressive crawling to keep models and retrieval fresh. For your site, the practical signal is that Anthropic is now fetching your pages heavily, so whether Claude can learn from and cite you increasingly depends on your letting ClaudeBot (train) and Claude-SearchBot (index) through.
Googlebot still tops the list, but the fast money is on Anthropic: ClaudeBot's one-month leap to #2 is the biggest single-crawler share gain Cloudflare has recorded. Crawl attention is following adoption.
How big has AI crawling become overall?
Large, and increasingly training-driven. Cloudflare reports that AI crawlers reached about 22% of all bot traffic in Q1 2026, up from a negligible share two years earlier. Now bigger than every bot category except traditional search-engine crawlers. And the mix has tilted toward training: roughly 80% of AI bot activity is training-purpose crawling, up from about 72% a year earlier.
The longer trend, year over year (Cloudflare, July 2024 to July 2025):
- GPTBot grew from ~4.7% to ~11.7% of overall crawl share.
- Meta-ExternalAgent jumped from ~0.9% to ~7.5%.
- Bytespider collapsed from ~14.1% to ~2.4% as ByteDance pulled back.
The takeaway: the set of bots visiting your site turns over fast, and a crawler that dominated last year can fade while a new one takes over. This is why a static robots.txt written once and forgotten drifts out of date.
Does heavy crawling mean heavy traffic back?
No, and the gap is stark. Cloudflare's crawl-to-referral data shows AI operators crawl far more than they send visitors: as of mid-2025, Anthropic crawled on the order of tens of thousands of pages for every referral it sent, and OpenAI on the order of a thousand-to-one. High crawl volume is a leading indicator of being learned and indexed, not a promise of clicks. The dynamic we cover in where AI crawls versus where it sends visitors. Judge a bot's crawling by whether it helps you get cited, not by referral traffic alone.
How should you use this data?
Treat it as a dated snapshot that informs access decisions, not a target:
- Decide per bot, not with one switch. Each crawler has a job. Train, index, or answer. Allow at least the index and answer bots that drive citations; the AI crawler user-agent directory and the train/index/answer lifecycle explain what each costs you if blocked.
- Don't let a CDN silently block the surging bots. A default "block all unknown bots" rule can cut off exactly the fast-growing crawlers like ClaudeBot. See is your CDN blocking AI crawlers.
- Watch your own logs, not just the industry chart. Which bots hit your site, and how often, is the signal that matters. The method in how to see which AI bots crawl your site.
- Revisit quarterly. Because shares move this fast, a crawl policy set once decays; review it as the leaderboard shifts.
The industry share chart tells you where crawler attention is heading in aggregate. What decides your AI visibility is which bots reach your pages and whether that access turns into citations for your brand. Measured on your own site and across engines over time. That is exactly what Buffy Intel is built to track.