Should you block AI bots? GPTBot, ClaudeBot and the robots.txt choice

Short answer

You can block AI training bots such as GPTBot, ClaudeBot and Google-Extended without disappearing from AI answers, because the AI companies use separate bots to search and cite websites. Block the search bots, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, and you can't be shown in those answers. For most UK businesses that want customers, the sensible middle ground is: allow search bots, decide on training.

Server racks in a dark room, with orange and blue network cables and rows of small green lights behind mesh doors
Photo: Taylor Vick on Unsplash

Key takeaways

  • OpenAI, Anthropic and Perplexity each run separate bots for training and for search. You can say no to one and yes to the other.
  • OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers. Blocking GPTBot only signals that your content shouldn't be used for training.
  • Google-Extended does not affect your place in Google Search. Google's AI Overviews are controlled by Googlebot and snippet settings, not by Google-Extended.
  • Bots that fetch a page because a person asked, such as ChatGPT-User and Perplexity-User, may not follow robots.txt at all.
  • Check your firewall and CDN too. Cloudflare changed its defaults to block AI crawlers, so some sites block them without the owner ever deciding to.

A client rings their web designer in a panic. They’ve read that AI companies are “stealing” small business websites, and they want every AI bot blocked by Friday. The designer pastes in a list of bot names from a forum, and a month later the business has quietly vanished from ChatGPT’s answers about their town. Nobody connects the two.

It’s an easy mistake, because the bots have confusing names and do very different jobs. Some read your site to train AI models. Others read it to answer a customer’s question right now and link back to you. Cloudflare’s data shows why owners feel short-changed by the first kind: in one week of June 2025, Anthropic’s bots requested about 70,900 pages for every visit they sent back (Cloudflare notes this may overstate it, because the Claude app doesn’t pass on where visitors came from). But blocking the wrong bot can cost you customers.

Here’s how to make the choice properly.

Should I block GPTBot?

You can, and it won’t take you out of ChatGPT search. OpenAI uses GPTBot to collect content that may be used to train its AI models. Its own documentation says that disallowing GPTBot “indicates a site’s content should not be used in training generative AI foundation models.”

ChatGPT’s search feature uses a different bot, OAI-SearchBot. OpenAI says plainly that “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”, and it recommends allowing it. It also says each setting is independent of the others.

So “block GPTBot” and “stay visible in ChatGPT” are not in conflict. The mistake is blocking OAI-SearchBot along with it, which is exactly what a copied “block all AI” list tends to do.

Blocking training is a business choice. Blocking search is usually an accident.

What’s the difference between training bots and search bots?

Training bots gather pages to help build future AI models. Search bots fetch pages so an assistant can answer questions today, usually with a link to the source. The big AI companies now publish separate names for each:

Company Training (safe to block) Search and citing (block = not shown) Fetches when a person asks
OpenAI GPTBot OAI-SearchBot ChatGPT-User
Anthropic ClaudeBot Claude-SearchBot Claude-User
Perplexity (none named) PerplexityBot Perplexity-User
Google Google-Extended Googlebot
Apple Applebot-Extended Applebot

A few details from each company’s own pages are worth knowing.

Anthropic says blocking ClaudeBot signals that your site’s future materials should be excluded from training. Blocking Claude-SearchBot stops it indexing your content for search, which it says may reduce your visibility in Claude’s search results. Blocking Claude-User stops Claude fetching your page when someone asks about it.

Perplexity says PerplexityBot is there to “surface and link websites in search results” and is not used to crawl content for AI models. Blocking it gains you nothing on training and costs you Perplexity’s answers.

Apple says Applebot-Extended doesn’t crawl at all. It only controls whether pages Applebot has already collected may be used to train Apple’s models, and blocking it doesn’t remove you from Apple’s search results.

What about Google’s AI Overviews?

Google works differently, and this trips a lot of people up. Google-Extended controls whether content Google crawls may be used to train future Gemini models and to ground answers in the Gemini apps and Vertex AI. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

AI Overviews and AI Mode are part of Google Search, so they come from pages that ordinary Googlebot has indexed. Google’s guide to AI features says a page only needs to be indexed and eligible to show with a snippet to appear as a supporting link, with “no additional technical requirements.” To limit what’s shown, it points to robots.txt rules for Googlebot and the usual snippet controls: nosnippet, data-nosnippet, max-snippet and noindex.

In practice: blocking Google-Extended is a training choice with no effect on your Google rankings. Blocking Googlebot to keep out of AI Overviews would take you out of Google altogether, which almost no business wants.

What does robots.txt actually do?

It’s a polite request, not a lock. Google’s own introduction says it’s “up to the crawler to obey” the rules, and that robots.txt is not a way to keep a page out of Google. For anything genuinely private, it recommends a password instead.

The AI companies’ pages back this up. OpenAI says that because ChatGPT-User acts when a person asks, “robots.txt rules may not apply.” Perplexity says Perplexity-User “generally ignores robots.txt rules” for the same reason. Anthropic says its bots honour robots.txt, and warns that blocking its IP addresses is not a reliable way to opt out, partly because it can stop the bot reading your robots.txt at all.

Two technical points catch people out:

  • A bot follows only its own group. Google explains that a crawler picks the group with the most specific matching name and ignores the others. If you add a section for GPTBot, it no longer reads your User-agent: * rules, so repeat any shared Disallow lines in each named group.
  • It works per subdomain. A robots.txt file only covers the host it sits on. If your shop lives on shop.yoursite.co.uk, that subdomain needs its own file.

Changes aren’t instant either. OpenAI says it can take about 24 hours after you update robots.txt for its search systems to adjust.

Could your site be blocking AI bots without you knowing?

Yes. Robots.txt is only one of the places a block can come from. Security plugins, hosting firewalls and content delivery networks can turn bots away before they ever reach your robots.txt.

Cloudflare, a widely used security and hosting service, announced in July 2025 that it was “changing the default to block AI crawlers” unless they pay. TechCrunch reported in July 2026 that from 15 September 2026 Cloudflare’s defaults would also block “mixed-use” crawlers from pages that carry ads, for new customers, new sites and all existing free customers. Whatever you think of those policies, the point for a business owner is simple: if your site uses Cloudflare, check its AI crawler settings rather than assume.

Which robots.txt should you use?

Pick the stance that fits your business, then copy the matching example. Each one keeps your normal rules for everything else. Swap in your own Disallow lines and sitemap address.

1. Allow everything (best for most businesses that want customers).

User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.yoursite.co.uk/sitemap.xml

2. AI search yes, AI training no (the middle ground).

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.yoursite.co.uk/sitemap.xml

Search bots such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot fall through to the * group, so they’re still welcome.

3. Block AI altogether (only if you’re sure).

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /

User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.yoursite.co.uk/sitemap.xml

This keeps you in Google and Bing but out of ChatGPT search, Claude’s search and Perplexity. That might suit a publisher whose writing is the product. For a plumber, an accountant or a café, it mostly means fewer people finding you.

Platform note: Shopify’s help pages say you change robots.txt by adding a robots.txt.liquid template to your theme, so edit that rather than pasting a whole file. Squarespace has a tick box, “Block known artificial intelligence crawlers”, in its Crawlers settings. Its list of 26 bots includes GPTBot, ClaudeBot and Google-Extended but not OAI-SearchBot, Claude-SearchBot or PerplexityBot, so it mostly leaves the main search bots alone. Squarespace also reminds owners that these are only requests and don’t remove anything already collected. On WordPress, an SEO plugin often manages the file, so make the change there.

How do you decide what’s right for you?

Ask one question: is my website how customers find me, or is my content the thing I sell?

If customers find you online, the search bots are working for you. Every time someone asks an assistant for “a good roofer in Hartlepool” or “the best Thai restaurant in Durham”, those bots are how your name can come up. Our guide to getting recommended by ChatGPT explains why being reachable is the first step.

Training is a genuine judgement call. Some owners are happy for AI to learn from their pages, on the basis that it helps assistants describe them accurately. Others don’t want their work used without payment. Both are reasonable, and the companies’ own documentation says blocking training bots is separate from search. Just remember it only affects future crawling, not models already trained.

How do you check it’s working?

Do these three things this week. None of them needs a developer.

  1. Read your robots.txt. Go to yoursite.co.uk/robots.txt and look for the bot names in the table above. Check each Disallow: / belongs to a bot you meant to block.
  2. Check your firewall or CDN. If you use Cloudflare, a security plugin or managed hosting, look for an “AI bots” or “AI crawler” setting and make sure it matches your robots.txt choice.
  3. Test from the outside. Run a free Spyglo check. It lists each major AI bot, says whether it can reach your site, and only marks you down for blocking the search bots, never for blocking training. If you’re on Pro, the robots.txt builder writes a file for whichever of the three stances you choose, with how-to guides for your platform.

Then give it a few weeks and ask ChatGPT, Claude and Perplexity the questions your customers would ask. If you’ve opened the door to the search bots, you’re giving yourself a fair chance of being in the answer. For more on what else decides that, see what GEO is.

The bottom line

“Should I block AI bots?” is really two questions. Should AI companies train on your content? That’s your call, and you can say no without losing search visibility. Should AI assistants be able to find and cite you? For almost every business that wants customers, yes. The worst outcome is the accidental one: a copied list or a default setting that shuts the search bots out while you think you’ve only said no to training.

See where your site stands

Spyglo checks your website the way Google and AI assistants see it, with an SEO score and a GEO score and the biggest thing to fix first. Free, in about a minute.

Check my site free

Questions people ask

Will blocking GPTBot stop ChatGPT recommending my business?

Not on its own. OpenAI uses GPTBot for training and a separate bot, OAI-SearchBot, for ChatGPT search. Blocking GPTBot signals that your content shouldn't be used for training; blocking OAI-SearchBot is what keeps you out of ChatGPT search answers.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search. AI Overviews draw on pages Googlebot has indexed, and Google's controls for them are robots.txt rules for Googlebot plus snippet settings such as nosnippet.

Does blocking AI bots remove my content from models already trained?

No. Anthropic, for example, says blocking ClaudeBot signals that your site's future materials should be left out of training. It works from the time the bot reads your new rules, not backwards.

Do all AI bots obey robots.txt?

No. Google points out that it's up to each crawler to obey the rules. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. If something must stay private, put it behind a password.

Sources

  1. OpenAI (2026). Overview of OpenAI crawlers. https://developers.openai.com/api/docs/bots
  2. Anthropic (2026). Does Anthropic crawl data from the web, and how can site owners block the crawler? https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  3. Google for Developers (2026). Google's common crawlers: Google-Extended. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
  4. Google Search Central (2025). AI features and your website. https://developers.google.com/search/docs/appearance/ai-features
  5. Google Search Central (2026). How Google interprets the robots.txt specification. https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
  6. Google Search Central (2025). Introduction to robots.txt. https://developers.google.com/search/docs/crawling-indexing/robots/intro
  7. Perplexity (2026). Perplexity crawlers. https://docs.perplexity.ai/guides/bots
  8. Apple (2026). About Applebot. https://support.apple.com/en-us/119829
  9. Cloudflare (2025). The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (Radar). https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/
  10. Cloudflare (2025). Content Independence Day: no AI crawl without compensation. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/
  11. Shopify Help Center (2026). Editing robots.txt.liquid. https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
  12. Squarespace Help Center (2026). Request that AI models exclude your site. https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site
  13. TechCrunch (2026). Cloudflare's new policy pushes AI companies to pay for publishers' content. https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/

Free tools to put this to work

More from Spyglo