Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
robots.txt group count
MCP serversrobots.txt group count
-
robots.txt Allow count, body discarded
MCP serversrobots.txt Allow count, body discarded
-
robots-txt-checker
MCP serversChecks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Robots.txt Policy Lint
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Hosted MCP: CSV/XLSX to JSON rows and text-PDF pages; private uploads or URLs, no API key.
-
Scrape-LE
MCP serversAnalyse robots.txt content and report whether a path may be crawled.
scrape-le-mcp@2.6.0 -
matchstick
MCP serversChecks a site build: sitemaps, structured data, robots.txt and what crawlers see.
@astralchemist/matchstick@0.1.0-alpha.4 -
Robots Inspect
MCP serversFetch a public robots.txt and return group counts only.
-
SEOAudit
MCP serversOn-page SEO analysis, HTTP header checks, broken link detection, and robots.txt parsing.
-
marsgeo
MCP serversGenerate SEO structured data: JSON-LD schema, robots.txt, sitemaps, hreflang, Open Graph, meta.
-
url2md
MCP serversPDFs, JavaScript pages and ordinary HTML as clean Markdown. Follows robots.txt per RFC 9309.
@url2md-io/mcp@0.1.5 -
PriceMoved
MCP serversWaitlist-stage pricing-page monitor: plans and early pricing, plus a PriceMovedBot robots.txt check.
-
Botcard
MCP serversCited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
-
Site Search Opener
MCP serversSEO and competitor research from sitemaps and feeds, robots.txt respected, Google link fallback.
site-search-opener-mcp@1.1.1 -
citable
MCP serversCan ChatGPT, Claude and Perplexity read and cite your site? robots.txt, edge blocks, llms.txt.
citable-cli@0.4.2 -
distill-mcp
MCP serversWeb pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.
hybridforgelab-distill-mcp@0.2.3 -
geoscore
MCP serversAI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.
@reneturcios/geoscore@0.1.0 -
distill-mcp
MCP serversWeb pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.
bytestay-distill-mcp@0.2.9 -
ai-visibility
MCP serversAI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
-
agent-ready-kit
MCP serversAgent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
-
stillindexed SEO checkers
MCP serversIndexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.