InnerVault

Robots.txt & AI Crawler Checker

Stale Official MCP Registry, dev.workers.quietworklabs.robots-check

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Open the source (opens robots-check.quietworklabs.workers.dev) https://robots-check.quietworklabs.workers.dev

Before you use it

  • Runs on someone else's server: what you send it leaves your machine.

What it is

MCP servers from dev.workers.quietworklabs.robots-check.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

Remote endpoint (streamable-http)

https://robots-check.quietworklabs.workers.dev/mcp

From the server's entry in the official MCP Registry

Related

  1. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  2. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  3. polite-crawler

    MCP servers

    A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

    Indexed Official MCP Registry, io.github.mark1russell7 Updated yesterday 1 note

    polite-crawler-mcp@0.7.6
  4. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  5. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  6. One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

    Stale Official MCP Registry, app.rowb.ai-visibility Updated 54 days ago 1 note