buffy.blog
Series · 13 articles

Crawlers & Technical Setup

The technical foundation of AI visibility: which AI crawlers visit you, how to let the right ones in, and how crawlability, CDN rules, page speed, and files like llms.txt decide whether you're reachable at all.

  1. 01Should you let AI crawlers index your site?GPTBot, ClaudeBot, Google-Extended, PerplexityBot. Your robots.txt decides whether AI engines can learn from and cite your brand. Here's how to think about blocking vs. allowing them.3 min read
  2. 02The complete AI crawler user-agent directoryEvery major AI crawler in one place. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more. What each one does (train, index, or answer), what blocking it costs you, and copy-paste robots.txt recipes.4 min read
  3. 03How to see which AI bots are crawling your siteYour server and CDN logs already record every AI crawler that visits. GPTBot, ClaudeBot, PerplexityBot and the rest. Here's how to read them to confirm you're reachable and catch silent blocks.2 min read
  4. 04Your CDN might be blocking the AI crawlers you wantYou opened robots.txt and added llms.txt, and AI engines still can't see you. The culprit is often one layer up: a CDN or WAF silently returning 403 to GPTBot, ClaudeBot, and PerplexityBot. Here's the 60-second check.3 min read
  5. 05Does page speed affect your AI visibility?Yes. Indirectly but really. Slow, heavy, JS-dependent pages are harder for AI crawlers to fetch, render, and extract, which can quietly keep you out of AI answers. Here's the mechanism and the fixes.2 min read
  6. 06Is llms.txt worth it? An honest takellms.txt is a tidy idea. A curated map of your site for AI. But as of 2026 the major engines don't broadly honour it. Here's the honest verdict: low-cost hygiene, not a visibility lever.2 min read
  7. 07Accessibility and AI: why your accessibility score predicts AI-parseabilityThe semantic structure that makes a page accessible to a screen reader is the same structure that lets an AI crawler segment, extract, and cite it. So your Lighthouse accessibility score doubles as a proxy for AI-readability. Here's how to use it.4 min read
  8. 08Do AI assistants render your JavaScript? What a 12-engine test foundA June-2026 test fed 12 AI assistants a page whose real answer was injected by JavaScript. Every major Western assistant. ChatGPT, Claude, Gemini, Perplexity, Copilot. Read only the raw HTML and missed it. Here's what that means for what you put in the markup.5 min read
  9. 09Do AI assistants respect robots.txt? A 12-engine testWhen a user pastes a blocked URL into a chatbot, does it obey your robots.txt? A June-2026 test put 12 assistants to the test. Only three honoured the block and declared an honest user-agent. Here's what that means for blocking versus being cited.4 min read
  10. 10Why Google says llms.txt is fundamentally flawedOn Search Off The Record in June 2026, Google's John Mueller explained why llms.txt can't make AI discover you: a file you write about yourself is a self-serving claim, and retrieval systems are built to distrust those. Here's the argument, and what actually earns AI trust.4 min read
  11. 11Why is Google showing markdown files in AI Overviews?In early July 2026, SEOs spotted Google AI Overviews pulling the raw contents of .md (markdown) files into citations. John Mueller called it 'unexpected' and said it doesn't mean Google treats markdown specially. Here's what actually happened, why it's a crawlability story not a ranking lever, and what to check on your own site.5 min read
  12. 12How Brave Search discovers your pages, and why it decides whether Claude can cite youBrave builds its index from anonymous signals contributed by opted-in browsers, not from anything you submit. No search console, no sitemap intake, no IndexNow. A page enters when about 20 opted-in users visit it or when one opted-in user sees it rank on another engine. Because Claude's live search leans on Brave, that pipeline increasingly decides whether Claude can cite you.7 min read
  13. 13Do AI crawlers follow JavaScript links? A 41-day, 27-bot experimentA mid-2026 field experiment split a 1,000-page site between plain HTML links and JavaScript-injected links, then logged 30,180 bot requests. Only Google's crawlers followed the JavaScript links; 26 of 27 bots. GPTBot, ClaudeBot, PerplexityBot and the rest. Found nothing behind them. Here's what that means for how AI discovers your pages.5 min read