Skip to content
How it worksPricingFree toolsWhat is GEO?Articles
Sign inCheck my site
How it worksPricingFree toolsWhat is GEO?ArticlesSign inCheck my site free
Facts checked 4 October 2026

AI crawler list: every AI bot, and what blocking it does

AI companies run three kinds of crawler: search crawlers that let their assistant find and cite you, fetchers that read a page when someone asks, and training crawlers that collect pages to train models. Block a search crawler and you disappear from that assistant’s answers. Block a training crawler and you don’t. Here are all 25, with what each one does and what blocking it costs you, from each company’s own documentation.

Want to see what your site allows right now? The free AI Crawler Checker reads your robots.txt bot by bot. For the trade-offs and ready-made robots.txt files, read should you block AI bots?

Search and citation crawlers

These build the indexes AI assistants search when they answer a question. Block one and you can’t be found or cited in that assistant’s answers.

User agentRun byWhat it doesFollows robots.txt?If you block it
OAI-SearchBotOpenAIChatGPT search results and citationsYesYou won’t appear in ChatGPT search answers
Claude-SearchBotAnthropicSearch index for Claude’s answersYesClaude can’t find your pages when it searches
PerplexityBotPerplexityPerplexity’s search index and citations (not training)YesYou won’t be cited in Perplexity answers
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeYesYou drop out of Google Search and its AI features
BingbotMicrosoftBing search and Microsoft CopilotYesYou drop out of Bing and Copilot answers
ApplebotAppleSpotlight, Siri and Safari searchYesYou drop out of Siri and Spotlight suggestions
Meta-WebIndexerMetaMeta AI searchYesMeta AI can’t find your pages
Amzn-SearchBotAmazonAlexa and Amazon search (not training)YesAlexa can’t use your pages in answers
DuckAssistBotDuckDuckGoLive fetches for DuckAssist answers (not training)Yes, within about 72 hoursDuckAssist can’t quote you
MistralAI-IndexMistralSearch index for Le Chat (not training)YesLe Chat can’t find your pages when it searches

Fetches a person asked for

These visit a page because someone asked the assistant to read it, for example by pasting your link. Some owners say robots.txt may not apply to them.

User agentRun byWhat it doesFollows robots.txt?If you block it
ChatGPT-UserOpenAIReads a page when a ChatGPT user asksOpenAI says robots.txt rules may not applyChatGPT may fail to open your page when asked
Claude-UserAnthropicReads a page when a Claude user asksYesClaude can’t open your page when asked
Perplexity-UserPerplexityReads a page when a Perplexity user asksPerplexity says it generally ignores robots.txtLittle effect, as it may not follow the rule
Meta-ExternalFetcherMetaReads a link a user shares with Meta AIMeta says it may bypass robots.txtLittle effect
Amzn-UserAmazonReads a page when an Alexa user asksAmazon says it may not fully follow itLittle effect
MistralAI-UserMistralReads a page when a Le Chat user asksYesLe Chat can’t open your page when asked

Training crawlers

These collect pages to train AI models. Blocking them is a fair choice and doesn’t remove you from AI search answers.

User agentRun byWhat it doesFollows robots.txt?If you block it
GPTBotOpenAITraining OpenAI’s modelsYesYour pages aren’t used to train OpenAI models. ChatGPT search is unaffected
ClaudeBotAnthropicTraining Anthropic’s modelsYes, and honours Crawl-delayYour pages aren’t used to train Claude. Claude search is unaffected
Google-ExtendedGoogleGemini training and Gemini Apps grounding (a rule token, not a separate crawler)YesDoesn’t affect Google Search or AI Overviews
Applebot-ExtendedAppleOpt-out from Apple’s model training (a rule token; Apple says it doesn’t crawl)YesDoesn’t affect Siri or Spotlight
Meta-ExternalAgentMetaAI training and indexingYesYour pages aren’t collected for Meta’s AI
CCBotCommon CrawlOpen web archive, widely used to train modelsYesKept out of the Common Crawl archive
AmazonbotAmazonAmazon products and AI trainingYes (ignores Crawl-delay)Kept out of Amazon’s crawl
MistralAI-TrainingMistralTraining Mistral’s modelsYesYour pages aren’t used to train Mistral models
BytespiderByteDancePresumed training (no official documentation found)Reported to ignore itMay have little effect; a firewall rule is more reliable

How do I block a training crawler but keep AI search?

Give the training bot its own group in robots.txt. Under the robots.txt standard (RFC 9309) a bot follows only the most specific group that names it, so this blocks GPTBot without touching OAI-SearchBot or Googlebot:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Two things catch people out. A bot with its own group ignores your User-agent: * rules, so repeat any shared rules inside it. And some firewalls and CDNs, including Cloudflare’s AI bot settings, block bots before robots.txt is ever read, so check those too.

Does blocking GPTBot keep me out of ChatGPT?

No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot only says your pages shouldn’t be used for training. To stay in ChatGPT’s answers, let OAI-SearchBot in.

Does Google-Extended affect AI Overviews?

No. Google-Extended controls Gemini training and grounding in Gemini Apps. Google’s AI Overviews and AI Mode use the normal Googlebot crawl, so they’re controlled by Googlebot rules and snippet settings like nosnippet.

Sources are linked on each company’s name. We re-check this list every quarter; spotted a change first? Tell us.

See your website the way search engines and AI assistants see it, then fix what's holding you back.

Email us

Product

Free website checkHow it worksPricingSign in

Free tools

GEO CheckerAI Crawler CheckerAI Visibility TestSchema Markup GeneratorGoogle Review Link and QR CodeAll free tools

Learn

What is GEO?ArticlesHow we scoreAI crawler listResearchThe two scoresQuestions

Company

PrivacyTermsCookiesCookiesAboutContact
© 2026 Spyglo, Middlesbrough.Spyglo only ever reads public web pages.