InnerVault

robots.txt group count

Stale Official MCP Registry, io.github.sadri-dridi

robots.txt group count

Open the source (opens github.com) https://github.com/sadri-dridi/named-mcp-utilities

Before you use it

  • Runs on someone else's server: what you send it leaves your machine.

What it is

MCP servers from io.github.sadri-dridi.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

Remote endpoint (streamable-http)

https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-count/mcp

From the server's entry in the official MCP Registry

Related

  1. Robots Inspect

    MCP servers

    Fetch a public robots.txt and return group counts only.

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  2. Doorman

    MCP servers

    Check which AI bots a website or a pasted robots.txt lets in, grouped by purpose.

    Stale Official MCP Registry, com.clawhard Updated 7 days ago 1 note

  3. tdm-reservation-lint

    MCP servers

    Finds the robots.txt lines that reserve nothing — retired crawler tokens, shadowed groups, rules abo

    Stale Official MCP Registry, io.github.jmshinhwa Updated 54 days ago 1 note

    @readystack/tdm-reservation-lint@1.0.5
  4. robots.txt Allow count, body discarded

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  5. robots-txt-checker

    MCP servers

    Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).

    Stale Official MCP Registry, io.github.ryuamarines Updated 18 days ago 1 note

  6. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note