robots.txt group count
robots.txt group count
Open the source (opens github.com) https://github.com/sadri-dridi/named-mcp-utilities
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from io.github.sadri-dridi.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-count/mcp
From the server's entry in the official MCP Registry
Related
-
Robots Inspect
MCP serversFetch a public robots.txt and return group counts only.
-
Doorman
MCP serversCheck which AI bots a website or a pasted robots.txt lets in, grouped by purpose.
-
tdm-reservation-lint
MCP serversFinds the robots.txt lines that reserve nothing — retired crawler tokens, shadowed groups, rules abo
@readystack/tdm-reservation-lint@1.0.5 -
robots.txt Allow count, body discarded
MCP serversrobots.txt Allow count, body discarded
-
robots-txt-checker
MCP serversChecks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).
-
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.