How do I know if GPTBot or ClaudeBot has visited my site?

Check your server access logs or CDN analytics and filter by user-agent. Each AI crawler announces itself. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, etc. If you see them returning HTTP 200, they're reaching you; if you see 403s (or don't see them at all), something is blocking them.

Can I trust the user-agent a bot reports?

Not blindly. User-agents can be spoofed. The major operators publish IP ranges (and some support reverse-DNS verification), so for anything important, confirm the request's IP really belongs to the operator rather than trusting the string alone.

How to see which AI bots are crawling your site

You don't have to guess whether AI engines can see your site. The evidence is already in your logs. Every well-behaved crawler announces itself by user-agent, and your server/CDN records each visit. Reading those logs is the fastest way to confirm you're actually reachable (and to catch a silent block before it costs you citations).

Where to look

Server access logs. Your web server (nginx, Apache) logs every request with its user-agent and the status code returned. Filter for AI crawler tokens.
CDN analytics. If you're behind a CDN (Cloudflare, Fastly, etc.), its dashboard usually classifies bot traffic. Often with a dedicated "AI crawlers / AI bots" view.
A quick spot-check without logs: impersonate a bot and read the status code directly (see below).

What to filter for

Search your logs for the crawler tokens you care about (the full set is in the AI crawler directory):

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Bytespider.

What the logs tell you

Three things worth reading:

Who's visiting: which engines crawl you at all. A bot that never appears can't cite you.
Status codes: 200 means you served the page; 403/401 means you're blocking that bot (often at the CDN, regardless of what robots.txt says. See is your CDN blocking AI crawlers?).
What and how often: which pages they fetch and at what frequency, mapped to the train / index / answer lifecycle (training bots crawl broadly; answer bots fetch a single page on demand).

The 30-second spot-check

No log access? Impersonate the bot and read the first response line:

curl -sI -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yourdomain.com | head -1

HTTP/2 200 = reachable. HTTP/2 403 = blocked. Repeat for the bots that matter.

Verify identity for anything important

A user-agent is self-reported and can be faked. For decisions that matter, confirm the request's IP belongs to the operator (they publish ranges; some support reverse-DNS) rather than trusting the string.

Your logs are the ground truth of AI reachability. "We allowed them in robots.txt" means nothing if the logs show 403s. The logs tell you what actually happened.

Reading logs is the manual version of a question worth answering continuously: are the engines reaching, crawling, and citing you, and did that change? Monitoring it across engines over time is what Buffy Intel is built to do.