Skip to content
How it worksPricingFree toolsWhat is GEO?Articles
Sign inCheck my site
How it worksPricingFree toolsWhat is GEO?ArticlesSign inCheck my site free
Last updated 26 October 2026

The SpygloResearch crawler

If you’ve seen SpygloResearch/1.0 in your website’s logs, it was us. Spyglo is a website checking service run from Unit 1C, Brighouse Road, Riverside Park, Middlesbrough, TS2 1RT.

What it does

  • It checks how well public business websites can be found by search engines and AI assistants, the same check anyone can run free at spyglo.com.
  • It reads your home page, robots.txt, sitemap, llms.txt and, if you have one, your contact page. That’s all: a handful of requests, once.
  • It never logs in, fills in forms or tries to reach anything that isn’t public.
  • We may use the result, with the business contact details your site publishes, to get in touch about improving your site. You can ask us not to at any time (see below).
  • It also gathers data for our published research, such as our study of UK council websites. Research results are only ever published for public bodies or as totals, never naming a private business.

How to stop it

We follow robots.txt. To keep the crawler out, add this to your site’s robots.txt:

User-agent: SpygloResearch
Disallow: /

A Disallow: / for all crawlers (User-agent: *) keeps it out too.

Asking us not to contact you

Email our support address with your website’s address. We’ll delete your contact details, mark your site so it’s never added again, and confirm by reply. More detail is in our privacy policy.

See your website the way search engines and AI assistants see it, then fix what's holding you back.

Email us

Product

Free website checkHow it worksPricingSign in

Free tools

GEO CheckerAI Crawler CheckerAI Visibility TestSchema Markup GeneratorGoogle Review Link and QR CodeAll free tools

Learn

What is GEO?ArticlesHow we scoreAI crawler listResearchThe two scoresQuestions

Company

PrivacyTermsCookiesCookiesAboutContact
© 2026 Spyglo, Middlesbrough.Spyglo only ever reads public web pages.