Our crawler
Our crawler identifies itself as:
schoolsitecheck-bot/1.0 (+https://schoolsitecheck.co.uk/bot/)
What it does
- Reads public pages of school websites in England to see which statutory documents are linked: up to 30 pages for a check someone runs on this site, and up to 40 for our scheduled checks.
- Makes at most one request per second to a site, and honours a longer
Crawl-delayin robots.txt. - Reads HTML pages only. It doesn’t download PDFs or other documents; it judges them by their links, titles and file names.
- Never logs in, fills in forms, or opens parent portals, search, calendar or login links.
How often
Whenever someone runs a free check on this site; for scheduled checks, roughly once a month per school website at most, and monthly for schools that pay for monitoring.
How to block it
It follows robots.txt. To block only our crawler, add:
User-agent: schoolsitecheck-bot Disallow: /
Or ask us to stop checking your website on the opt-out page; that also stops our emails. Problems? Email [email protected].