oetio
Book a call →
oetio

Explainer

How AI search engines work: retrieve, rank, write, cite

How AI search engines work: each one retrieves pages from a web index, ranks them, has a language model write an answer, and cites its sources. Only a few companies run their own index. Google, Microsoft's Bing, Perplexity, and Brave built their own indexes. Most assistants, including ChatGPT and Claude, read someone else's. Whichever index a product reads decides which of your pages it can ever cite.

Published 2026-07-22Updated 2026-10-0421 primary6 min read

Key takeaways

  • Every AI answer sits on a web index underneath it. Only Google, Bing, Perplexity, and Brave run their own.[6][18]
  • ChatGPT reads Bing's index.[11] Claude's web search shows strong evidence, not confirmation, of reading Brave's.[17]
  • Across 100,000 identical prompts, ChatGPT and Perplexity shared only 11% of their cited sources.[21]
In this article

What are the four steps of an AI search engine?#

Every AI search answer follows the same path. First the engine retrieves candidate pages. For many questions it rewrites your question into several searches first, a step Google calls query fan-out.[2] Then it ranks the pages it found. Then a language model writes an answer from the top passages. Last, the engine cites the sources it used. Each step can only work with what the step before it passed along, and the first step can only retrieve what the web index stored.

Step 1RetrievePull candidate pagesStep 2RankSort the best matchesStep 3WriteModel writes the answerStep 4CiteLink the sourcesWeb index (Google, Bing, Perplexity or Brave) built by a crawler
Our own diagram of the four steps. The index layer is what the next sections cover: who runs it decides what can be retrieved.

What is a search index?#

A search engine has three jobs. A crawler visits pages on the web and copies their text. An index stores what the crawler found, in a database built to be searched fast. Then a ranking system pulls the best matches out of that index and serves them.[1] That pipeline is the search engine, and it existed long before generative AI. An AI answer engine adds one more step: a language model reads the top results and writes them up as an answer, with citations. But the model can only cite a page the ranking stage surfaced, the ranking stage can only surface a page the index stored, and the index never stores a page its crawler was blocked from fetching. Eligibility is decided at the bottom of the stack, not by how smart the model is.

Which index does each engine use?#

Building a web index is expensive: crawling tens of billions of pages, storing them, and crawling them again constantly so the data does not rot. Very few companies do it.

  1. Google: AI Overviews and Gemini on Google's own index

    AI Overviews run on Google's existing Search systems, ranking systems, and the Knowledge Graph, and pull results straight from Google's own index.[2]AI Mode splits one question into many searches at once (query fan-out), and Google's Deep Search can issue hundreds of searches to build a cited report in minutes.[3] Gemini grounds answers the same way through the Google Search API.[4]Google-Extended lets a site opt out of AI training and grounding, but Google states this does not affect a site's inclusion or ranking in ordinary Search, so plain Googlebot still governs whether AI Overviews can cite you.[5]

  2. Bing: one index behind Copilot, DuckDuckGo, and ChatGPT

    Microsoft 365 Copilot sends your query to the Bing search service to ground its answer.[7] Microsoft calls the Bing global index a continuously refined map of the web built over decades, now sold to outside companies as Microsoft Web IQ.[8] It retired the public Bing Search APIs on August 11, 2025 and moved customers to Grounding with Bing Search instead.[9] Microsoft states plainly that Bing powers a wide range of AI tools, including Yahoo, DuckDuckGo, Microsoft Start, and ChatGPT.[10] DuckDuckGo confirms it from its own side: it crawls with DuckDuckBot, but sources its links and images largely from Bing.[20] If Copilot gets your company wrong, see how to fix incorrect Microsoft Copilot answers.

  3. Perplexity: the one AI company that built its own index

    Perplexity did not rent. Its search index tracks over 200 billion unique URLs, on a crawling system it describes as running at exabyte scale with tens of thousands of CPUs.[6] That is a fourth independent web index, built specifically to answer questions. It behaves like an island: winning there says little about any other engine.

  4. ChatGPT: a GPT-4o model in front of Bing

    OpenAI says ChatGPT search uses outside search providers, running on a version of GPT-4o tuned for search.[11] Its help center names the providers directly, Bing and Shopify, and explains that ChatGPT rewrites your question into one or more targeted queries before sending them out.[12]No source shows OpenAI running a general web index of its own. Two crawlers matter: GPTBot feeds training, and OAI-SearchBot decides what can appear in ChatGPT's search answers. Blocking OAI-SearchBot removes a site from those answers; blocking GPTBot does not.[13]

  5. Claude: not documented, but the evidence points to Brave

    Anthropic's documentation gives Claude direct access to live web content and names no search provider anywhere.[14] The tool launched on the API on May 7, 2025 at $10 per 1,000 searches.[15] Anthropic runs three crawlers: ClaudeBot for training, Claude-User for live requests, and Claude-SearchBot, built to improve search results.[16]The evidence for Brave is strong: Anthropic's subprocessor list added Brave Search one day before web search launched, Claude's tool schema leaked a BraveSearchParams field, and independent testers matched Claude's citations to Brave's results.[17] Brave itself sells access to a fully independent index of over 30 billion pages, independent of Bing since April 2023.[18][19] Anthropic has never confirmed any of it.

EngineIndex it readsStatus
Google AI Overviews, AI Mode, GeminiGoogle's own index[2][4]Documented by Google
Microsoft 365 CopilotBing[7]Documented by Microsoft
ChatGPT searchBing and other outside providers[11][12]Documented by OpenAI
PerplexityIts own index, over 200 billion URLs[6]Documented by Perplexity
Claude web searchNot named; evidence points to Brave[17]Not confirmed by Anthropic
Each row cites the company's own documentation, except Claude, where the Brave link rests on a subprocessor listing and independent tests.

How do you get cited?#

  1. Ask, do not assume

    Ask each engine the questions your buyers ask, three times. That shows you which index, Google, Bing, Perplexity, or Brave, already has you, and where you are missing.

  2. Match the crawler to its purpose

    OAI-SearchBot, not GPTBot, controls ChatGPT search inclusion.[13] Claude-User and Claude-SearchBot gate Claude's live answers, separately from ClaudeBot, which only handles training.[16] A blanket bot block meant to opt out of training routinely deletes a site from search answers by accident.

  3. Get named where the index already looks

    An index can only cite pages it already crawled and stored. Get your name onto the review sites, lists, and forums each engine already ranks for your buyer's question.

  4. Put the answer first

    Structure pages so the direct answer sits in the first two sentences, then verify by asking each engine again after the change.

What this means for you#

None of this is theoretical. Across 100,000 identical prompts, ChatGPT and Perplexity shared only 11% of their cited sources.[21] Different index, different crawler rules, different result. The work is concrete and per engine: let the right crawler in, be present in the indexes your buyers use, and confirm it by asking, repeatedly, and measuring.

Noetio's AI SEO services do this measuring and fixing for companies that AI engines leave out of their answers. We test what ChatGPT, Perplexity, Gemini, and Google already say about you, find which index is missing your pages, and fix it with crawler rules, mentions, and pages that answer the question first. See how this looked for one company in why your company is not in ChatGPT. For one engine in depth, read how ChatGPT search works, how Perplexity search works or how Claude web search works. For what to change on your pages, see LLM SEO.

FAQ

How do AI search engines work?
They run four steps. They retrieve candidate pages from a web index, rank those pages, write an answer with a language model, and cite the sources they used. The index sits underneath: a page that is not in the index cannot be retrieved, ranked or cited.
Which search index does ChatGPT use?
OpenAI says ChatGPT search uses outside search providers, and its help center names Bing and Shopify. No source we found shows OpenAI running a general web index of its own.
Do AI search engines use Google?
Some do. Google's own AI Overviews, AI Mode and Gemini use Google's index. ChatGPT reads Bing, Perplexity built its own index, and Claude's provider is not named by Anthropic, though evidence points to Brave.
How do I stop an AI search engine from citing my site?
Block the crawler that feeds that engine's search. For ChatGPT search that is OAI-SearchBot, not GPTBot, which only handles training. Blocking the wrong bot can remove you from answers by accident.

Not cited in the AI engines your buyers use? Run the free check.

Book a call →Run the free check →

Sources

  1. Crawl, index, and serve definitions and “the Google index, which is a large database”: Google, “In-depth guide to how Google Search works”.
  2. AI Overviews use “our existing Search systems… and the Google Knowledge Graph” and “results from our index”; the “query fan-out technique”: Google, “AI Overviews and AI Mode in Search”, May 2025.
  3. Deep Search “can issue hundreds of searches… a fully-cited report in just minutes”: Google, “AI Mode in Google Search: Updates from Google I/O 2025”, May 20, 2025.
  4. Grounding with Google Search connects Gemini to real-time web content via Google Search: Google, “Grounding with Google Search” (Gemini API).
  5. “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal”: Google Search Central, “Google's common crawlers”.
  6. “Search index tracks over 200 billion unique URLs”; “exabyte-scale index and crawling apparatus”: Perplexity Research, “Architecting and Evaluating an AI-First Search API”, September 25, 2025.
  7. Microsoft 365 Copilot “may fetch information from the Bing search service” and “sends [a query] to the Bing search service” to ground responses: Microsoft Learn, “Data, privacy, and security for web search in Microsoft 365 Copilot”.
  8. “The Bing global index… a continuously refined representation of the web, built over decades”: Microsoft, “Announcing Microsoft Web IQ”, June 22, 2026.
  9. Bing Search APIs “will be retired on August 11, 2025”; migrate to “Grounding with Bing Search as part of Azure AI Agents”: Microsoft Learn, “Bing Search APIs Retiring on August 11, 2025”, announced May 15, 2025.
  10. “Bing powers a wide range of AI and search experiences, including Yahoo, DuckDuckGo, Microsoft Start, and AI tools like ChatGPT and Inflection.ai”: Microsoft Bing Webmaster Blog, June 17, 2025.
  11. ChatGPT search uses outside search providers, running a version of GPT-4o tuned for search: OpenAI, “Introducing ChatGPT search”, October 31, 2024.
  12. ChatGPT “rewrites your query… sends those providers” and names “Bing (Microsoft)” and “Shopify” as third-party search providers: OpenAI Help Center, “ChatGPT Search”.
  13. GPTBot crawls content “used in training”; “OAI-SearchBot is used to surface websites in search results”; opting out of OAI-SearchBot means a site “will not be shown in ChatGPT search answers”: OpenAI, “Overview of OpenAI Crawlers”.
  14. The web search tool gives Claude “direct access to real-time web content”; no provider is named: Anthropic, “Web search tool” documentation.
  15. Web search launched on the Anthropic API at “$10 per 1,000 searches”: Anthropic, “Web search on the Anthropic API”, May 7, 2025.
  16. Three crawlers: ClaudeBot (training), Claude-User (live requests), and Claude-SearchBot, which “navigates the web to improve search result quality”: Anthropic Support, “Does Anthropic crawl data from the web?”.
  17. Brave Search added to Anthropic's subprocessor list on March 19, 2025 (one day before launch); a BraveSearchParams field in Claude's tool schema; Claude citations matching Brave results: Simon Willison, “Anthropic's new subprocessor: Brave Search”, March 21, 2025.
  18. Brave Search API is “powered by Brave Search, a completely independent index of the Web” and “not a scraper that simply uses bots to query Google or Bing”: Brave, “Brave Search API”.
  19. Brave's “independent index of over 30 billion pages”; full search independence from Bing reached in April 2023: Brave, “Brave Search independence”, April 27, 2023.
  20. DuckDuckGo runs “our own crawler (DuckDuckBot)” but sources traditional links and images “largely… from Bing”: DuckDuckGo, “Sources” help page.
  21. ChatGPT and Perplexity share 11.0% of cited sources across 100,000 prompts: Profound, July 2025.

Every architecture claim quotes the named company's own documentation, blog, or help center, retrieved and verified on July 22, 2026. The link between Claude and Brave is labeled as evidenced, not confirmed, because Anthropic has not stated its provider; it rests on Anthropic's own subprocessor disclosure plus independent, reproducible testing. Study figures are cited as published by their sources.

If you want this done for you

SEO and AI SEOGet found when they search, and named when they ask an AIWhat it costsScoped on a call, in writing the same day