# AI crawler reference

Source: https://noetio.com/docs/crawlers

This table is the classifier Noetio runs in production to attribute AI crawler
traffic for customers. It is the canonical list we check audit findings
against. Three purposes matter, and they have different consequences:

- **search-index**: the bot builds the retrieval index an engine answers from.
  Blocking it removes you from that engine's answers.
- **user-fetch**: the bot fetches your page live because a user's question
  triggered it. Blocking it breaks citations at the moment they would happen.
- **training**: the bot collects training data. Blocking it does not remove
  you from today's answers; it is a policy choice, not a visibility fix.

## The crawlers

| User agent | Operator | Purpose | Verification |
| --- | --- | --- | --- |
| GPTBot | OpenAI | training | published IP list (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | search index | published IP list (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | user fetch | published IP list (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | training | published IP list (claude.com/crawling/bots.json) |
| Claude-User | Anthropic | user fetch | published IP list (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | search index | published IP list (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | search index | published IP list |
| Perplexity-User | Perplexity | user fetch | published IP list |
| Google-Extended | Google | training and Gemini grounding control | reverse DNS |
| Applebot | Apple | search index | reverse DNS |
| Amazonbot | Amazon | search index | reverse DNS |
| Bingbot | Microsoft | search index | reverse DNS |
| Bytespider | ByteDance | training | none published, UA only |
| meta-externalagent | Meta | training | none published, UA only |
| Meta-ExternalFetcher | Meta | user fetch | none published, UA only |
| MistralAI-User | Mistral | user fetch | none published, UA only |
| GrokBot | xAI | training | none published, UA only |

Bots marked "UA only" publish no IP ranges or reverse DNS scheme, so any
traffic claiming those names cannot be verified. Treat log entries from them
accordingly.

Two non-obvious rules that decide real findings:

- **Google AI Overviews uses regular Googlebot.** Blocking Google-Extended
  does not remove you from AI Overviews; it removes you from Gemini app
  grounding and future training. If you block Googlebot, you have bigger
  problems than GEO.
- **ChatGPT's web answers need OAI-SearchBot and ChatGPT-User, not GPTBot.**
  Sites that block everything containing "GPT" to opt out of training also
  cut themselves out of ChatGPT search citations.

## robots.txt template

A starting point that keeps you citable everywhere while leaving the
training-data decision to you:

```text
# Answer-time and index crawlers: blocking these removes you from AI answers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training crawlers: allowing these is a policy choice, not a visibility fix.
# Replace Allow with Disallow if you opt out of training data collection.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /
```

Bingbot, Googlebot and Applebot follow your existing SEO rules; do not add AI
specific blocks for them.

## Check what a crawler actually sees

Fetch your page the way an AI crawler does and look at the HTML that comes
back without JavaScript:

```bash
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  -sL https://yourdomain.com/pricing | grep -c "your product name"
```

If the count is 0 but the page shows the name in a browser, engines are
reading an empty shell. That is a rendering finding, the most common class in
our audits, and the fix is server side rendering for the affected routes.

To verify that traffic in your logs claiming to be GPTBot is real, resolve
the source IP against the vendor's published list:

```bash
curl -s https://openai.com/gptbot.json | grep -o '"[0-9./]*"'
```

An IP outside those ranges is not GPTBot, whatever its user agent says.

## Related

- [Fix categories](/docs/fixes): what an audit prescribes when access is broken.
- [Re-run verification](/docs/verification): how a fix is proven afterwards.
- The [measurement protocol](/methodology) documents how we verify crawler
  identity in customer traffic.