InnerVault

Robots.txt Policy Lint

Stale Official MCP Registry, dev.workers.pathwren.www

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

Open the source (opens pathwren.workers.dev) https://pathwren.workers.dev/c/mcp-registry-official/mcp-robots.html

Before you use it

  • Runs on someone else's server: what you send it leaves your machine.

What it is

MCP servers from dev.workers.pathwren.www.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

Remote endpoint (streamable-http)

https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots

From the server's entry in the official MCP Registry

Related

  1. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  2. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  3. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  4. robots.txt group count

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  5. robots.txt Allow count, body discarded

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  6. robots-txt-checker

    MCP servers

    Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).

    Stale Official MCP Registry, io.github.ryuamarines Updated 18 days ago 1 note