AI Crawler Checker — Robots.txt Policy for Six Bots
Check declared robots.txt permissions for six AI crawlers at URL paths
Open the source (opens apify.com) https://apify.com/hermeskd/ai-crawler-access-check
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from io.github.kriton-dev.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check
From the server's entry in the official MCP Registry
Related
-
Unrendered: AI Crawler Checker
MCP serversAI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
unrendered-mcp@0.1.4 -
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
AI Crawler Access Checker
MCP serversShows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
-
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
-
AI Visibility Checker
MCP serversAI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.
https://github.com/krisdiallo/ecom-agent/releases/download/v1.5.3/aivis.mcpb@1.5.3 -
AI Bot Access Checker
MCP serversChecks if GPTBot, ClaudeBot, PerplexityBot, Googlebot and 9 more AI crawlers can reach a page.