InnerVault

AI Crawler Checker — Robots.txt Policy for Six Bots

Stale Official MCP Registry, io.github.kriton-dev

Check declared robots.txt permissions for six AI crawlers at URL paths

Open the source (opens apify.com) https://apify.com/hermeskd/ai-crawler-access-check

Before you use it

  • Runs on someone else's server: what you send it leaves your machine.

What it is

MCP servers from io.github.kriton-dev.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

Remote endpoint (streamable-http)

https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check

From the server's entry in the official MCP Registry

Related

  1. AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

    Stale Official MCP Registry, io.github.missiongrowth Updated 26 days ago 1 note

    unrendered-mcp@0.1.4
  2. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  3. Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

    Stale Official MCP Registry, io.github.Nero-Engine Updated 15 days ago 1 note

  4. Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.

    Stale Official MCP Registry, io.github.stellalytics Updated 3 days ago 1 note

  5. AI Visibility Checker

    MCP servers

    AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.

    Stale Official MCP Registry, io.github.krisdiallo Updated 2 months ago 1 note

    https://github.com/krisdiallo/ecom-agent/releases/download/v1.5.3/aivis.mcpb@1.5.3
  6. AI Bot Access Checker

    MCP servers

    Checks if GPTBot, ClaudeBot, PerplexityBot, Googlebot and 9 more AI crawlers can reach a page.

    Stale Official MCP Registry, io.seopaslaugos Updated 4 days ago 1 note