AI crawler list: every AI bot, and what blocking it does
AI companies run three kinds of crawler: search crawlers that let their assistant find and cite you, fetchers that read a page when someone asks, and training crawlers that collect pages to train models. Block a search crawler and you disappear from that assistant’s answers. Block a training crawler and you don’t. Here are all 25, with what each one does and what blocking it costs you, from each company’s own documentation.
Want to see what your site allows right now? The free AI Crawler Checker reads your robots.txt bot by bot. For the trade-offs and ready-made robots.txt files, read should you block AI bots?
Search and citation crawlers
These build the indexes AI assistants search when they answer a question. Block one and you can’t be found or cited in that assistant’s answers.
| User agent | Run by | What it does | Follows robots.txt? | If you block it |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search results and citations | Yes | You won’t appear in ChatGPT search answers |
Claude-SearchBot | Anthropic | Search index for Claude’s answers | Yes | Claude can’t find your pages when it searches |
PerplexityBot | Perplexity | Perplexity’s search index and citations (not training) | Yes | You won’t be cited in Perplexity answers |
Googlebot | Google Search, including AI Overviews and AI Mode | Yes | You drop out of Google Search and its AI features | |
Bingbot | Microsoft | Bing search and Microsoft Copilot | Yes | You drop out of Bing and Copilot answers |
Applebot | Apple | Spotlight, Siri and Safari search | Yes | You drop out of Siri and Spotlight suggestions |
Meta-WebIndexer | Meta | Meta AI search | Yes | Meta AI can’t find your pages |
Amzn-SearchBot | Amazon | Alexa and Amazon search (not training) | Yes | Alexa can’t use your pages in answers |
DuckAssistBot | DuckDuckGo | Live fetches for DuckAssist answers (not training) | Yes, within about 72 hours | DuckAssist can’t quote you |
MistralAI-Index | Mistral | Search index for Le Chat (not training) | Yes | Le Chat can’t find your pages when it searches |
Fetches a person asked for
These visit a page because someone asked the assistant to read it, for example by pasting your link. Some owners say robots.txt may not apply to them.
| User agent | Run by | What it does | Follows robots.txt? | If you block it |
|---|---|---|---|---|
ChatGPT-User | OpenAI | Reads a page when a ChatGPT user asks | OpenAI says robots.txt rules may not apply | ChatGPT may fail to open your page when asked |
Claude-User | Anthropic | Reads a page when a Claude user asks | Yes | Claude can’t open your page when asked |
Perplexity-User | Perplexity | Reads a page when a Perplexity user asks | Perplexity says it generally ignores robots.txt | Little effect, as it may not follow the rule |
Meta-ExternalFetcher | Meta | Reads a link a user shares with Meta AI | Meta says it may bypass robots.txt | Little effect |
Amzn-User | Amazon | Reads a page when an Alexa user asks | Amazon says it may not fully follow it | Little effect |
MistralAI-User | Mistral | Reads a page when a Le Chat user asks | Yes | Le Chat can’t open your page when asked |
Training crawlers
These collect pages to train AI models. Blocking them is a fair choice and doesn’t remove you from AI search answers.
| User agent | Run by | What it does | Follows robots.txt? | If you block it |
|---|---|---|---|---|
GPTBot | OpenAI | Training OpenAI’s models | Yes | Your pages aren’t used to train OpenAI models. ChatGPT search is unaffected |
ClaudeBot | Anthropic | Training Anthropic’s models | Yes, and honours Crawl-delay | Your pages aren’t used to train Claude. Claude search is unaffected |
Google-Extended | Gemini training and Gemini Apps grounding (a rule token, not a separate crawler) | Yes | Doesn’t affect Google Search or AI Overviews | |
Applebot-Extended | Apple | Opt-out from Apple’s model training (a rule token; Apple says it doesn’t crawl) | Yes | Doesn’t affect Siri or Spotlight |
Meta-ExternalAgent | Meta | AI training and indexing | Yes | Your pages aren’t collected for Meta’s AI |
CCBot | Common Crawl | Open web archive, widely used to train models | Yes | Kept out of the Common Crawl archive |
Amazonbot | Amazon | Amazon products and AI training | Yes (ignores Crawl-delay) | Kept out of Amazon’s crawl |
MistralAI-Training | Mistral | Training Mistral’s models | Yes | Your pages aren’t used to train Mistral models |
Bytespider | ByteDance | Presumed training (no official documentation found) | Reported to ignore it | May have little effect; a firewall rule is more reliable |
How do I block a training crawler but keep AI search?
Give the training bot its own group in robots.txt. Under the robots.txt standard (RFC 9309) a bot follows only the most specific group that names it, so this blocks GPTBot without touching OAI-SearchBot or Googlebot:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /Two things catch people out. A bot with its own group ignores your User-agent: * rules, so repeat any shared rules inside it. And some firewalls and CDNs, including Cloudflare’s AI bot settings, block bots before robots.txt is ever read, so check those too.
Does blocking GPTBot keep me out of ChatGPT?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot only says your pages shouldn’t be used for training. To stay in ChatGPT’s answers, let OAI-SearchBot in.
Does Google-Extended affect AI Overviews?
No. Google-Extended controls Gemini training and grounding in Gemini Apps. Google’s AI Overviews and AI Mode use the normal Googlebot crawl, so they’re controlled by Googlebot rules and snippet settings like nosnippet.
Sources are linked on each company’s name. We re-check this list every quarter; spotted a change first? Tell us.