Robots.txt & AI Crawler Checker
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Open the source (opens robots-check.quietworklabs.workers.dev) https://robots-check.quietworklabs.workers.dev
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from dev.workers.quietworklabs.robots-check.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://robots-check.quietworklabs.workers.dev/mcp
From the server's entry in the official MCP Registry
Related
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Check declared robots.txt permissions for six AI crawlers at URL paths
-
polite-crawler
MCP serversA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
polite-crawler-mcp@0.7.6 -
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.