Robots.txt Policy Lint
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
Open the source (opens pathwren.workers.dev) https://pathwren.workers.dev/c/mcp-registry-official/mcp-robots.html
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from dev.workers.pathwren.www.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots
From the server's entry in the official MCP Registry
Related
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
robots.txt group count
MCP serversrobots.txt group count
-
robots.txt Allow count, body discarded
MCP serversrobots.txt Allow count, body discarded
-
robots-txt-checker
MCP serversChecks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).