robots.txt Allow count, body discarded
robots.txt Allow count, body discarded
Open the source (opens github.com) https://github.com/sadri-dridi/named-mcp-utilities
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from io.github.sadri-dridi.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp
From the server's entry in the official MCP Registry
Related
-
robots-txt-checker
MCP serversChecks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).
-
robots.txt group count
MCP serversrobots.txt group count
-
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Robots.txt Policy Lint
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key