AI crawls almost everything on your site, but sends visitors to almost none of it. A mid-2026 analysis of more than 560,000 AI crawl requests across 74 websites found homepages get roughly 15 times the crawl attention of other pages, articles get read heavily but rarely clicked, and 47% of all pages earned zero referral visits at all. The headline for marketers: crawl volume tells you AI can reach your pages; it says almost nothing about which pages AI will send people to. Those are two different maps, and they barely overlap.
This is a data explainer, not a new dataset. It sits alongside our AI search statistics reference. The figures come from Orbit Media's study by Andy Crestodina, published early July 2026, built on Cloudflare AI Crawl Control exports (the "Most Crawled Paths" and "AI Referral Traffic" reports) covering 560,695 crawl requests and 446,267 referrals across 74 sites (27 of them on paid tiers with referral tracking) and 21,212 URL paths. Every figure below is attributed, dated, and hedged: it is one vendor's dataset, skewed toward B2B and services sites, so read the patterns as directional.
Which pages does AI crawl the most?
Homepages, overwhelmingly. Adjusted for site size. A homepage is one page competing with dozens or hundreds of others. Homepages drew about 15x the AI crawl attention of any other page type. Nothing else came close; every other category clustered below its proportional share.
| Page type | Relative AI crawl attention (1.0 = proportional) |
|---|---|
| Homepage | ~15x |
| Service / product | 0.85x |
| Articles / resources | 0.82x |
| About | 0.80x |
| Contact | 0.77x |
| Pricing | 0.68x |
| Case studies | 0.56x |
| Docs (PDFs) | 0.56x |
Source: Orbit Media / Cloudflare AI Crawl Control, 74 sites, mid-2026 (median crawl intensity, normalised for site size; single-vendor, directional). The clear read: if you want AI to learn one thing about your brand, the homepage is where it looks first, which is why the homepage now does a discovery job it didn't a year ago.
Site size mattered too. Page count alone predicted total crawl volume almost perfectly. A 0.86 correlation, explaining about 73% of the variation in total AI requests. More pages meant more surface area to match against more prompts. But it wasn't destiny: some small sites drew far more attention than their size predicted, likely from stronger brands and better-optimised pages.
Why do crawled pages not turn into visits?
Because reading a page and recommending a click are separate decisions. This is the study's most important finding, and Orbit Media gave it a name: the Dark Library Effect: pages AI reads heavily and summarises, but rarely sends a visitor to. Articles are the clearest case. They get crawled and cited, then the answer satisfies the user in place, and no click follows. It's the same zero-click dynamic that reshaped classic search, now applied to AI.
Comparing each page type's share of referrals against its share of crawls exposes the gap directly:
| Page type | Referral share minus crawl share (points) |
|---|---|
| Homepage | +10.4 |
| Service / product | +2.2 |
| About | −0.1 |
| Contact | −0.8 |
| Docs (PDFs) | −2.6 |
| Articles / resources | −8.7 |
Source: Orbit Media / Cloudflare, 27 sites with referral tracking, mid-2026 (directional). A positive number means a page type sends more traffic than its crawl volume predicts; a negative number means it's read more than it's clicked. Homepages and service pages over-deliver on traffic; articles under-deliver by the widest margin. Service and product pages earned roughly three times more AI-referral traffic per page than a typical article. Even though the typical site in the study had more articles than service pages.
The Dark Library Effect: AI reads your articles far more than it sends anyone to them. In this dataset, 47% of all pages earned zero referral visits. Crawled, summarised, and never clicked.
This is not an argument to stop publishing. It's the same lesson our freshness and citation work keeps surfacing: content that gets crawled and cited builds the model's knowledge of your brand even when it drives no click. Crestodina's own conclusion is that content marketing stays valuable without referral traffic. It trains AI, earns citations, and supports the sale in ways a click counter never sees. The mistake is applying click-through goals to pages that were never going to win the click. That's also why heavy crawling with few citations is a leading indicator, not a failure: the crawl comes first.
Does page depth change how much traffic AI sends?
Sharply. AI crawlers will find pages buried deep in a folder structure. Crawl budget stretches far, but they rarely recommend them. Orbit Media called this an "architecture tax," and the drop-off with depth is steep.
| Folder depth | AI-referral intensity (1.0 = proportional) |
|---|---|
One level deep (/services/) |
0.91 |
Two levels (/services/seo/) |
0.68 |
Three levels (/services/seo/local/) |
0.25 |
| Four or more levels | 0.05 |
Source: Orbit Media / Cloudflare, mid-2026 (median referral intensity by depth; directional). A page three folders deep earned about a quarter of the traffic its footprint predicted; at four levels, close to zero. The study is careful, and so are we, that this is correlation, not causation: moving a page shallower won't automatically win it traffic, and you shouldn't reshuffle your whole site over one dataset. But if you're planning a new structure, keeping your most important pages within a click or two of the homepage is a defensible default.
What should marketers actually do with this?
Treat crawl and referral as two separate measurements, and match the page to the job. AI visits a site for two different reasons. To fill in background knowledge (training crawlers) and to answer a live question (search-agent crawlers), and neither guarantees a click. Practical moves from the data:
- Make the homepage a real briefing. It gets the most crawl attention and over-delivers on traffic, so state plainly who you are, what you do, and your differentiators in extractable text, not a slogan over a hero image.
- Invest in service and product pages if you want AI referrals that may convert; they earn far more traffic per page than articles.
- Keep publishing articles, but judge them right. Expect citations and brand-training value, not direct clicks. Measure citations, not just referral clicks.
- Don't bury your best pages. Deep pages get crawled but rarely recommended.
- Remember AI traffic hides. Much of it lands in analytics as direct traffic, so the referral counts you see understate the real total, and AI visitors often convert better than the raw numbers suggest.
The deeper point is that "we get crawled a lot" and "AI recommends us" are different claims that need different evidence. Watching where AI actually crawls, cites, and sends visitors across engines, and separating the Dark Library Effect from real recommendation. Is exactly what Buffy Intel is built to measure. Questions: [email protected].