AI crawler reference
Every AI crawler that matters for GEO: what each does, how to verify it is real, and a robots.txt template that keeps you citable.
This table is the classifier Noetio runs in production to attribute AI crawler traffic for customers. It is the canonical list we check audit findings against. Three purposes matter, and they have different consequences:
- search-index: the bot builds the retrieval index an engine answers from. Blocking it removes you from that engine's answers.
- user-fetch: the bot fetches your page live because a user's question triggered it. Blocking it breaks citations at the moment they would happen.
- training: the bot collects training data. Blocking it does not remove you from today's answers; it is a policy choice, not a visibility fix.
The crawlers
| User agent | Operator | Purpose | Verification |
|---|---|---|---|
| GPTBot | OpenAI | training | published IP list (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | search index | published IP list (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | user fetch | published IP list (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | training | published IP list (claude.com/crawling/bots.json) |
| Claude-User | Anthropic | user fetch | published IP list (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | search index | published IP list (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | search index | published IP list |
| Perplexity-User | Perplexity | user fetch | published IP list |
| Google-Extended | training and Gemini grounding control | reverse DNS | |
| Applebot | Apple | search index | reverse DNS |
| Amazonbot | Amazon | search index | reverse DNS |
| Bingbot | Microsoft | search index | reverse DNS |
| Bytespider | ByteDance | training | none published, UA only |
| meta-externalagent | Meta | training | none published, UA only |
| Meta-ExternalFetcher | Meta | user fetch | none published, UA only |
| MistralAI-User | Mistral | user fetch | none published, UA only |
| GrokBot | xAI | training | none published, UA only |
Bots marked "UA only" publish no IP ranges or reverse DNS scheme, so any traffic claiming those names cannot be verified. Treat log entries from them accordingly.
Two non-obvious rules that decide real findings:
- Google AI Overviews uses regular Googlebot. Blocking Google-Extended does not remove you from AI Overviews; it removes you from Gemini app grounding and future training. If you block Googlebot, you have bigger problems than GEO.
- ChatGPT's web answers need OAI-SearchBot and ChatGPT-User, not GPTBot. Sites that block everything containing "GPT" to opt out of training also cut themselves out of ChatGPT search citations.
- Groups do not inherit.
User-agent: *is only the fallback: once a crawler finds a group naming it, that group applies and the*group does not. So the moment you add aUser-agent: GPTBotblock, everyDisallowyou had under*stops applying to GPTBot. This is the most common way a site opens its private paths while trying to become more citable, and it is silent: the file still reads as though those paths are protected.
robots.txt template
A starting point that keeps you citable everywhere while leaving the training-data decision to you.
Replace /admin/ below with whatever your User-agent: * group already
disallows, and make sure it appears in every group. A group with an
Allow and no Disallow is permitted everywhere, so a group that omits your
private paths grants that crawler access to them.
Written out one crawler per group below because it is easier to read and to
edit later. If you would rather not repeat yourself, stack the User-agent
lines and give them one shared rule list — crawlers with identical policies
can share a group, and the rules apply to all of them.
User-agent: *
Allow: /
Disallow: /admin/
# Answer-time and index crawlers: blocking these removes you from AI answers.
# Each group repeats the Disallow list because groups do not inherit from *.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
User-agent: ChatGPT-User
Allow: /
Disallow: /admin/
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
User-agent: Perplexity-User
Allow: /
Disallow: /admin/
User-agent: Claude-User
Allow: /
Disallow: /admin/
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
# Training crawlers: allowing these is a policy choice, not a visibility fix.
# Replace Allow with Disallow if you opt out of training data collection.
User-agent: GPTBot
Allow: /
Disallow: /admin/
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
User-agent: Google-Extended
Allow: /
Disallow: /admin/If a group has nothing to exclude, keep the Disallow: line empty rather
than dropping it, so the next person editing the file can see the decision
was made rather than forgotten.
Bingbot, Googlebot and Applebot follow your existing SEO rules; do not add AI specific blocks for them.
Check what a crawler actually sees
Fetch your page the way an AI crawler does and look at the HTML that comes back without JavaScript:
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
-sL https://yourdomain.com/pricing | grep -c "your product name"If the count is 0 but the page shows the name in a browser, engines are reading an empty shell. That is a rendering finding, the most common class in our audits, and the fix is server side rendering for the affected routes.
To verify that traffic in your logs claiming to be GPTBot is real, resolve the source IP against the vendor's published list:
curl -s https://openai.com/gptbot.json | grep -o '"[0-9./]*"'An IP outside those ranges is not GPTBot, whatever its user agent says.
Related
- Fix categories: what an audit prescribes when access is broken.
- Re-run verification: how a fix is proven afterwards.
- The measurement protocol documents how we verify crawler identity in customer traffic.