School Site Check

Our crawler

Our crawler identifies itself as:

schoolsitecheck-bot/1.0 (+https://schoolsitecheck.co.uk/bot/)

What it does

  • Reads public pages of school websites in England to see which statutory documents are linked: up to 30 pages for a check someone runs on this site, and up to 40 for our scheduled checks.
  • Makes at most one request per second to a site, and honours a longer Crawl-delay in robots.txt.
  • Reads HTML pages only. It doesn’t download PDFs or other documents; it judges them by their links, titles and file names.
  • Never logs in, fills in forms, or opens parent portals, search, calendar or login links.

How often

Whenever someone runs a free check on this site; for scheduled checks, roughly once a month per school website at most, and monthly for schools that pay for monitoring.

How to block it

It follows robots.txt. To block only our crawler, add:

User-agent: schoolsitecheck-bot
Disallow: /

Or ask us to stop checking your website on the opt-out page; that also stops our emails. Problems? Email [email protected].