AI crawler reference
Every AI crawler that matters for GEO, what each one does, how to verify it is real, and a robots.txt template that keeps you citable. Maintained from Noetio's production crawler classifier.
This table is the classifier Noetio runs in production to attribute AI crawler traffic for customers. It is the canonical list we check audit findings against. Three purposes matter, and they have different consequences:
- search-index: the bot builds the retrieval index an engine answers from. Blocking it removes you from that engine's answers.
- user-fetch: the bot fetches your page live because a user's question triggered it. Blocking it breaks citations at the moment they would happen.
- training: the bot collects training data. Blocking it does not remove you from today's answers; it is a policy choice, not a visibility fix.
The crawlers
| User agent | Operator | Purpose | Verification |
|---|---|---|---|
| GPTBot | OpenAI | training | published IP list (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | search index | published IP list (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | user fetch | published IP list (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | training | published IP list (claude.com/crawling/bots.json) |
| Claude-User | Anthropic | user fetch | published IP list (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | search index | published IP list (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | search index | published IP list |
| Perplexity-User | Perplexity | user fetch | published IP list |
| Google-Extended | training and Gemini grounding control | reverse DNS | |
| Applebot | Apple | search index | reverse DNS |
| Amazonbot | Amazon | search index | reverse DNS |
| Bingbot | Microsoft | search index | reverse DNS |
| Bytespider | ByteDance | training | none published, UA only |
| meta-externalagent | Meta | training | none published, UA only |
| Meta-ExternalFetcher | Meta | user fetch | none published, UA only |
| MistralAI-User | Mistral | user fetch | none published, UA only |
| GrokBot | xAI | training | none published, UA only |
Bots marked "UA only" publish no IP ranges or reverse DNS scheme, so any traffic claiming those names cannot be verified. Treat log entries from them accordingly.
Two non-obvious rules that decide real findings:
- Google AI Overviews uses regular Googlebot. Blocking Google-Extended does not remove you from AI Overviews; it removes you from Gemini app grounding and future training. If you block Googlebot, you have bigger problems than GEO.
- ChatGPT's web answers need OAI-SearchBot and ChatGPT-User, not GPTBot. Sites that block everything containing "GPT" to opt out of training also cut themselves out of ChatGPT search citations.
robots.txt template
A starting point that keeps you citable everywhere while leaving the training-data decision to you:
# Answer-time and index crawlers: blocking these removes you from AI answers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Training crawlers: allowing these is a policy choice, not a visibility fix.
# Replace Allow with Disallow if you opt out of training data collection.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /Bingbot, Googlebot and Applebot follow your existing SEO rules; do not add AI specific blocks for them.
Check what a crawler actually sees
Fetch your page the way an AI crawler does and look at the HTML that comes back without JavaScript:
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
-sL https://yourdomain.com/pricing | grep -c "your product name"If the count is 0 but the page shows the name in a browser, engines are reading an empty shell. That is a rendering finding, the most common class in our audits, and the fix is server side rendering for the affected routes.
To verify that traffic in your logs claiming to be GPTBot is real, resolve the source IP against the vendor's published list:
curl -s https://openai.com/gptbot.json | grep -o '"[0-9./]*"'An IP outside those ranges is not GPTBot, whatever its user agent says.
Related
- Fix categories: what an audit prescribes when access is broken.
- Re-run verification: how a fix is proven afterwards.
- The measurement protocol documents how we verify crawler identity in customer traffic.