Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
-
distill-mcp
MCP serversWeb pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.
bytestay-distill-mcp@0.2.9 -
ai-visibility
MCP serversAI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
-
agent-ready-kit
MCP serversAgent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
-
stillindexed SEO checkers
MCP serversIndexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.
-
tdm-reservation-lint
MCP serversFinds the robots.txt lines that reserve nothing — retired crawler tokens, shadowed groups, rules abo
@readystack/tdm-reservation-lint@1.0.5 -
AI Visibility Checker
MCP serversAI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.
https://github.com/krisdiallo/ecom-agent/releases/download/v1.5.3/aivis.mcpb@1.5.3 -
polite-crawler
MCP serversA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
polite-crawler-mcp@0.7.6 -
robotstxt-ai
MCP serversAnalyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
robotstxt-ai-mcp@1.0.1 -
QueryRev AI Readiness
MCP serversChecks a site for AI crawler access, robots.txt and llms.txt. Runs locally, no telemetry.
@queryrev/ai-readiness-mcp@1.0.0 -
Doorman
MCP serversCheck which AI bots a website or a pasted robots.txt lets in, grouped by purpose.
-
SEOClave
MCP serversGoogle keyword data for Latin America plus technical SEO, robots.txt, DNS and AI-crawler checks.
-
llms.txt 檢查工具
MCP servers檢查網站 llms.txt 與 AI 爬蟲 robots.txt 設定,並自動產生 llms.txt。
-
Page Fetch
MCP serversRead one public web page as clean Markdown with its links, after a robots.txt check.
-
GEO SEO Records
MCP serversGets a website ready for AI assistants: robots.txt fixed, llms.txt drafted (access pass).
-
RobotsGate
MCP serversGenerate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
-
WebRead
MCP serversTurn any public web page into clean Markdown for LLMs; respects robots.txt. By Crypto Money Maker
-
ai-seo
MCP serversMCP server for AI-SEO auditing: schema, robots.txt, llms.txt, citation scoring, and AEO/GEO rewrites
@automatelab/ai-seo-mcp@0.1.1 -
NetLens
MCP serversRead any web page as clean, ad-stripped Markdown past robots.txt and bot blocks, plus web search.
netlens-mcp@0.2.3 -
Agent Crawl Readiness
MCP serversBounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
-
web-analytics-mcp
MCP serversWeb Analitiği MCP sunucusu — in-house on-page SEO denetimi (robots.txt + SSRF korumalı crawl),…
-
Citehound
MCP serversScan any domain for AI readiness and generate llms.txt, robots.txt and schema. Free, read-only.
-
GEO & SEO Readiness (Lite)
MCP serversChecks robots.txt for AI search vs training crawlers, llms.txt, JSON-LD, canonicals and sitemaps.
ghcr.io/alicexixi85/geo-seo-mcp-pro:0.1.0 -
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.