The SpygloResearch crawler
If you’ve seen SpygloResearch/1.0 in your website’s logs, it was us. Spyglo is a website checking service run from Unit 1C, Brighouse Road, Riverside Park, Middlesbrough, TS2 1RT.
What it does
- It checks how well public business websites can be found by search engines and AI assistants, the same check anyone can run free at spyglo.com.
- It reads your home page, robots.txt, sitemap, llms.txt and, if you have one, your contact page. That’s all: a handful of requests, once.
- It never logs in, fills in forms or tries to reach anything that isn’t public.
- We may use the result, with the business contact details your site publishes, to get in touch about improving your site. You can ask us not to at any time (see below).
- It also gathers data for our published research, such as our study of UK council websites. Research results are only ever published for public bodies or as totals, never naming a private business.
How to stop it
We follow robots.txt. To keep the crawler out, add this to your site’s robots.txt:
User-agent: SpygloResearchDisallow: /
A Disallow: / for all crawlers (User-agent: *) keeps it out too.
Asking us not to contact you
Email our support address with your website’s address. We’ll delete your contact details, mark your site so it’s never added again, and confirm by reply. More detail is in our privacy policy.