Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
robots.txt group count
MCP serversrobots.txt group count
-
robots.txt Allow count, body discarded
MCP serversrobots.txt Allow count, body discarded
-
robots-txt-checker
MCP serversChecks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Robots.txt and Sitemap Checker
Prompts--- name: robots-txt-and-sitemap-checker description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards…
-
Robots.txt Policy Lint
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Seo-Promt-Master
PromptsOpen-source SEO & GEO audit: a zero-dependency CLI (npx), an agent skill for Claude Code, Cursor, Codex, Gemini & Copilot, 47 SEO prompts and a cited Google/Bing/AI-search knowledge base. Checks robots.txt, sitemaps, canonicals, hreflang, structured data, Core Web Vitals and AI crawlers — then fixes them.
git clone https://github.com/umutxyp/Seo-Promt-Master.git -
Hosted MCP: CSV/XLSX to JSON rows and text-PDF pages; private uploads or URLs, no API key.
-
CitedBy
Public APIsWhich AI crawlers can read a site: robots.txt and server answer for each
-
Scrape-LE
MCP serversAnalyse robots.txt content and report whether a path may be crawled.
scrape-le-mcp@2.6.0 -
matchstick
MCP serversChecks a site build: sitemaps, structured data, robots.txt and what crawlers see.
@astralchemist/matchstick@0.1.0-alpha.4 -
Robots Inspect
MCP serversFetch a public robots.txt and return group counts only.
-
TidyTools
Public APIsWeb page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)
-
SEOAudit
MCP serversOn-page SEO analysis, HTTP header checks, broken link detection, and robots.txt parsing.
-
marsgeo
MCP serversGenerate SEO structured data: JSON-LD schema, robots.txt, sitemaps, hreflang, Open Graph, meta.
-
url2md
MCP serversPDFs, JavaScript pages and ordinary HTML as clean Markdown. Follows robots.txt per RFC 9309.
@url2md-io/mcp@0.1.5 -
PriceMoved
MCP serversWaitlist-stage pricing-page monitor: plans and early pricing, plus a PriceMovedBot robots.txt check.
-
Botcard
MCP serversCited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
-
Site Search Opener
MCP serversSEO and competitor research from sitemaps and feeds, robots.txt respected, Google link fallback.
site-search-opener-mcp@1.1.1 -
citable
MCP serversCan ChatGPT, Claude and Perplexity read and cite your site? robots.txt, edge blocks, llms.txt.
citable-cli@0.4.2 -
distill-mcp
MCP serversWeb pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.
hybridforgelab-distill-mcp@0.2.3 -
geoscore
MCP serversAI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.
@reneturcios/geoscore@0.1.0
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.