Check my site
Free tool · no sign-up

AI Crawler Checker

Find out in seconds whether ChatGPT, Claude, Perplexity, Google and Bing are allowed to read your website. We read your robots.txt the way the bots do, show the exact line that lets each one in or keeps it out, and check the page settings that can keep you out of AI answers.

What this checks

Two kinds of AI bot visit your website, and they do different jobs.

  • Search and citation bots fetch pages so an assistant can answer a question and link to you: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot and Perplexity-User (Perplexity), plus Googlebot and Bingbot, whose indexes feed Google’s AI Overviews and Microsoft Copilot. Block these and you can’t be found or quoted.
  • Training bots collect pages to train future AI models: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot. Blocking them is a fair choice and doesn’t stop you appearing in AI answers today.

Most checkers lump the two together. We keep them apart, because blocking the wrong group is the commonest mistake we see.

Page settings that keep you out of AI answers

A robots.txt that lets everyone in isn’t the whole story. These settings, in a page’s robots meta tag or the X-Robots-Tag header, also matter:

  • noindex: the page won’t be listed in search at all.
  • nosnippet: Google says it stops the page being used as a direct source for AI Overviews and AI Mode.
  • max-snippet: caps how much text Google can use from the page.
  • noarchive and nocache: Bing’s guidelines (February 2026) say noarchive keeps content out of Copilot answers, and nocache limits Copilot to your address, title and snippet.
  • data-nosnippet: text inside these parts of a page won’t be quoted.

robots.txt says yes, but are you still blocked?

Possibly. Since 1 July 2025 Cloudflare has blocked AI crawlers by default on new domains, and other firewalls can do the same. Those blocks check the bot’s real network address, so no outside tool can test them honestly (we don’t pretend to be a bot). If we spot that your site runs through Cloudflare, we’ll tell you where to check the setting.

How to fix a blocked bot

  1. Find the rule we show next to the blocked bot. It’s the exact line in your robots.txt.
  2. Remove that Disallow line, or add a group for the bot with Allow: /.
  3. On WordPress, robots.txt usually comes from your SEO plugin (Yoast: Tools, File editor; Rank Math: General Settings, Edit robots.txt). On Shopify, edit robots.txt.liquid in your theme code.
  4. Run this check again. Changes count as soon as the bots next read your robots.txt.

Google retired its own robots.txt tester in 2023, so a quick outside test like this one is the easiest way to be sure.

Sources

By the Spyglo Team · Facts checked 10 October 2026

See everything that’s holding your site back

This tool checks one thing. Spyglo’s free check runs 28 checks on your site and gives you an SEO score, a GEO score and the fixes, in under a minute. No card, no sign-up for your first check.

More free tools