Research

How AI search engines actually work: the index machines behind ChatGPT, Claude, Perplexity, and Google

An AI answer is only as good as the web index underneath it, and running a web index is rare. Only a handful of companies crawl the open web and operate their own: Google, Microsoft (Bing), Perplexity, and Brave. Most assistants, including ChatGPT and Claude, are a language model bolted onto someone else's search engine. Which index a product uses decides which of your pages it can ever cite. Every claim below links to a primary source.

Published 2026-07-2221 sources, first-party docs13 min read

Michael NefedovFounder

Ask five AI assistants the same question and you get five answers built from five different views of the web. That is not because the models disagree. It is because underneath each one sits a search engine, and those search engines are not the same, are not all owned by the company whose logo you see, and in several cases are the same rented index wearing different clothes. If you want your pages cited in AI answers, the machine you actually have to satisfy is the index, not the chatbot. Here is what each of the major engines crawls, stores, and ranks, and who really owns it.

What a search index actually is#

Google describes its own search in three stages. Crawling: “Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.” Indexing: “Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.” Then serving: when someone searches, the system returns relevant results.[1] That pipeline, a crawler feeding an index that a ranking stage queries, is the search engine. It existed long before generative AI.

An AI answer engine adds one stage on the end: a language model that reads the top-ranked results and writes them up in prose with citations. That last step is the part users see and the part that carries a brand name. But the model cannot cite a page the ranking stage did not surface, and the ranking stage cannot surface a page the index never stored, and the index never stores a page the crawler was blocked from fetching. Eligibility is decided at the bottom of the stack. This is why the interesting question is never “how smart is the model,” it is “whose index is it reading, and are you in it.”

Who owns an index, and who rents one#

Building a web index is expensive: you crawl tens of billions of pages, parse them, store them at exabyte scale, and re-crawl constantly so the data does not rot. Very few organizations do it. On the evidence of their own documentation, four run a large independent web index today: Google with Googlebot,[2] Microsoft with bingbot,[8] Perplexity, which built its own,[6]and Brave, whose search API is “powered by Brave Search, a completely independent index of the Web.”[18] Everyone else in AI search calls one of these indexes through an API or a partnership.

The clearest proof that renting is the norm is that Microsoft sells it. In May 2025 Microsoft announced that the Bing Search APIs would “be retired on August 11, 2025” and pushed customers toward “Grounding with Bing Search as part of Azure AI Agents.”[9]By June 2026 that had become Microsoft Web IQ, pitched as “a search engine for AI systems” built on “the Bing global index… a continuously refined representation of the web, built over decades.”[8]An entire product line exists to let other companies answer questions from Bing's index without building one. Below, each engine in turn.

Google: AI Overviews and Gemini on Google's own index#

Google is the one case where the answer engine and the index belong to the same company, and Google says so directly. AI Overviews “use a customized Gemini model, which works in tandem with our existing Search systems, like our quality and ranking systems and the Google Knowledge Graph,” and are “designed to carry out traditional search tasks, like identifying relevant, high-quality results from our index.”[2] There is no third-party index in the loop. It is Gemini reading the same Google index that powers the ten blue links.

What is new is how the query is handled. AI Mode “uses a query fan-out technique, issuing multiple related searches concurrently across subtopics and multiple data sources and then brings those results together.”[2]Google's I/O 2025 post puts a number on the heavy version: Deep Search “can issue hundreds of searches, reason across disparate pieces of information, and create an expert-level fully-cited report in just minutes.”[3]One question you type becomes many queries against Google's index, which multiplies the specific, long-tail phrasings where a focused page can win.

The crawler control here is worth knowing precisely. Google-Extended lets a site opt out of having its content used to train Gemini and for grounding, but Google states that “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”[5] Because AI Overviews are documented as drawing on the ordinary Search index and ranking systems,[2] the crawler that governs your presence there is plain Googlebot, the same one that has always governed Search.

Microsoft: Bing's index behind Copilot, and behind others#

Microsoft Copilot runs on Bing. In its documentation Microsoft states that when web search is on, “Microsoft 365 Copilot… may fetch information from the Bing search service,” and that “Copilot generates a search query that it sends to the Bing search service” to ground its responses.[7]Bing's crawler, bingbot, builds the index; Copilot is the writer on top of it, the same owner on both ends as Google.

The part most people miss is how much else runs on that one index. Microsoft states outright that “Bing powers a wide range of AI and search experiences, including Yahoo, DuckDuckGo, Microsoft Start, and AI tools like ChatGPT and Inflection.ai.”[10]DuckDuckGo confirms it from its own side: it runs its own crawler, DuckDuckBot, but its traditional links and images are “largely source[d] from Bing.”[20]When Bing's index changes, a whole tier of products that most users think of as separate changes with it.

Perplexity: the AI-native company that built its own index#

Perplexity is the exception that proves the pattern. Rather than keep renting, it built its own search infrastructure. Its engineering deep dive states the “search index tracks over 200 billion unique URLs” on an “exabyte-scale index and crawling apparatus” comprising “tens of thousands of CPUs and hundreds of terabytes of RAM.”[6] That is a genuine fourth Western web index, built by an AI company specifically to feed answers.

Because Perplexity owns the whole stack, it has published more about its retrieval and ranking than anyone, including how it decomposes pages into individually ranked spans. We took that architecture apart on its own in a companion piece, how Perplexity search works, according to Perplexity. The short version for this map: when you optimize for Perplexity, you are optimizing for an index no other engine shares, which is why winning there says little about anywhere else.

ChatGPT and OpenAI: a model in front of Bing#

OpenAI does not hide it. The launch post says ChatGPT search “leverages third-party search providers, as well as content provided directly by our partners,” and that the search model is “a fine-tuned version of GPT-4o.”[11]OpenAI's current help center names the providers: when you ask a question, ChatGPT “rewrites your query into one or more targeted queries that it sends those providers,” and points you to the privacy policies of “Bing (Microsoft)” and “Shopify” for how they process those queries.[12] Microsoft, from its side, lists ChatGPT among the AI tools Bing powers.[10] No primary source shows OpenAI operating its own general web index; the retrieval is rented.

OpenAI does run its own crawlers, and they have distinct jobs. Its documentation is explicit: GPTBot “is used to crawl content that may be used in training our generative AI foundation models,” while “OAI-SearchBot is used to surface websites in search results in ChatGPT's search features.”[13]The practical line is sharp: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”[13]Blocking GPTBot is a training decision; blocking OAI-SearchBot removes you from ChatGPT's answers. Many sites confuse the two and quietly delete themselves from ChatGPT while trying only to opt out of training. We take the query rewriting, the Bing dependency, and the crawler trap apart in full in how ChatGPT search works.

Anthropic and Claude: an unnamed provider that the evidence says is Brave#

Anthropic is the most opaque of the group. Its web search documentation describes the tool only functionally, giving Claude “direct access to real-time web content,” and names no provider anywhere.[14]The web search tool launched on the Anthropic API in May 2025 at “$10 per 1,000 searches.”[15] On the identity of the search backend, the company says nothing.

The evidence nonetheless points hard at Brave. Anthropic's own subprocessor list added “Brave Search” on March 19, 2025, one day before Claude's in-app web search went live, and independent testers found a BraveSearchParams field in Claude's tool schema and reproduced Brave's exact results as Claude's citations.[17]Brave, for its part, sells access to “a completely independent index of the Web” of “over 30 billion pages.”[18][19] That is as strong as circumstantial evidence gets, a legal disclosure plus a leaked parameter name plus matching results, but Anthropic has never confirmed it, so we report it as evidenced, not established.

One more nuance: Anthropic is not purely a renter. Its crawler documentation lists three separate bots, ClaudeBot for “collecting web content that could potentially contribute to” training, Claude-User for live user requests, and Claude-SearchBot, which “navigates the web to improve search result quality.”[16] So Anthropic does some of its own search crawling even while the live web search tool appears to lean on Brave. If you want to be eligible everywhere Claude can reach, those are the agents to allow. The full case, including the evidence trail for the Brave backend, is in how Claude's web search works.

What this means for getting cited#

The single practical consequence of the whole map is this: there is no “AI” to optimize for. There are three or four indexes, and each product is a front end to one of them. Optimizing for ChatGPT is, mechanically, optimizing to be in Bing's index and surfaced by OAI-SearchBot. Optimizing for Claude is, on the evidence, optimizing for Brave's index. Optimizing for AI Overviews is optimizing for Google's index. Perplexity is its own island. A page can be a strong citation in one and absent from another for no reason other than which index each one reads.

That is not a theory. Across 100,000 identical prompts, ChatGPT and Perplexity shared only 11 percent of their cited sources.[21]Different index, different crawler rules, different result. So the work is concrete and per-engine: let the right crawler in for each one, be present in the indexes your buyers' assistants actually read, and structure pages so the retrieval stage can find and rank the passage that answers the question. Then verify it, because the only way to know whether you are in a given engine's index is to ask that engine, repeatedly, and measure.

One last point, because it is the part that decides outcomes. Measuring your AI visibility is easy now: a dozen platforms score you across engines and stop there. Improving it is a different, harder job, changing the pages each engine reads with the specific levers that move citations, and confirming the change held. Noetio does that part, the fixes and the proof they worked, not just the score. The sourced side-by-side against the monitoring platforms is on the compare page; the full methodology shows how the lift is verified. Knowing how the engines work, which this post gave you, is step one. Being the source they cite is the harder step.

Sources

1. Crawl, index, and serve definitions and “the Google index, which is a large database”: Google, “In-depth guide to how Google Search works”.
2. AI Overviews use “our existing Search systems… and the Google Knowledge Graph” and “results from our index”; the “query fan-out technique”: Google, “AI Overviews and AI Mode in Search”, May 2025.
3. Deep Search “can issue hundreds of searches… a fully-cited report in just minutes”: Google, “AI Mode in Google Search: Updates from Google I/O 2025”, May 20, 2025.
4. Grounding with Google Search connects Gemini to real-time web content via Google Search: Google, “Grounding with Google Search” (Gemini API).
5. “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal”: Google Search Central, “Google's common crawlers”.
6. “Search index tracks over 200 billion unique URLs”; “exabyte-scale index and crawling apparatus”: Perplexity Research, “Architecting and Evaluating an AI-First Search API”, September 25, 2025.
7. Microsoft 365 Copilot “may fetch information from the Bing search service” and “sends [a query] to the Bing search service” to ground responses: Microsoft Learn, “Data, privacy, and security for web search in Microsoft 365 Copilot”.
8. “The Bing global index… a continuously refined representation of the web, built over decades”: Microsoft, “Announcing Microsoft Web IQ”, June 22, 2026.
9. Bing Search APIs “will be retired on August 11, 2025”; migrate to “Grounding with Bing Search as part of Azure AI Agents”: Microsoft Learn, “Bing Search APIs Retiring on August 11, 2025”, announced May 15, 2025.
10. “Bing powers a wide range of AI and search experiences, including Yahoo, DuckDuckGo, Microsoft Start, and AI tools like ChatGPT and Inflection.ai”: Microsoft Bing Webmaster Blog, June 17, 2025.
11. ChatGPT search “leverages third-party search providers”; the search model is “a fine-tuned version of GPT-4o”: OpenAI, “Introducing ChatGPT search”, October 31, 2024.
12. ChatGPT “rewrites your query… sends those providers” and names “Bing (Microsoft)” and “Shopify” as third-party search providers: OpenAI Help Center, “ChatGPT Search”.
13. GPTBot crawls content “used in training”; “OAI-SearchBot is used to surface websites in search results”; opting out of OAI-SearchBot means a site “will not be shown in ChatGPT search answers”: OpenAI, “Overview of OpenAI Crawlers”.
14. The web search tool gives Claude “direct access to real-time web content”; no provider is named: Anthropic, “Web search tool” documentation.
15. Web search launched on the Anthropic API at “$10 per 1,000 searches”: Anthropic, “Web search on the Anthropic API”, May 7, 2025.
16. Three crawlers: ClaudeBot (training), Claude-User (live requests), and Claude-SearchBot, which “navigates the web to improve search result quality”: Anthropic Support, “Does Anthropic crawl data from the web?”.
17. Brave Search added to Anthropic's subprocessor list on March 19, 2025 (one day before launch); a BraveSearchParams field in Claude's tool schema; Claude citations matching Brave results: Simon Willison, “Anthropic's new subprocessor: Brave Search”, March 21, 2025.
18. Brave Search API is “powered by Brave Search, a completely independent index of the Web” and “not a scraper that simply uses bots to query Google or Bing”: Brave, “Brave Search API”.
19. Brave's “independent index of over 30 billion pages”; full search independence from Bing reached in April 2023: Brave, “Brave Search independence”, April 27, 2023.
20. DuckDuckGo runs “our own crawler (DuckDuckBot)” but sources traditional links and images “largely… from Bing”: DuckDuckGo, “Sources” help page.
21. ChatGPT and Perplexity share 11.0% of cited sources across 100,000 prompts: Profound, July 2025.

Every architecture claim quotes the named company's own documentation, blog, or help center, retrieved and verified on July 22, 2026. The Claude-to-Brave link is labeled as evidenced, not confirmed, because Anthropic has not stated its provider; it rests on Anthropic's own subprocessor disclosure plus independent, reproducible testing. Study figures are cited as published by their sources.