Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git -
Crawlerpeek
MCP serversCheck whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
crawlerpeek@0.3.1 -
ai-crawler-policy
MCP serversCloudflare Workers MCP server: ai-crawler-policy
-
graphlit-mcp-server
MCP serversModel Context Protocol (MCP) Server for Graphlit Platform
git clone https://github.com/graphlit/graphlit-mcp-server.git -
Open SEO Crawler
MCP serversSelf-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
https://github.com/puneetindersingh/open-seo-crawler/releases/download/mcp-v1.0.0/open-seo-crawler-mcp.mcpb@1.0.0 -
polite-crawler
MCP serversA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
polite-crawler-mcp@0.7.6 -
Unrendered: AI Crawler Checker
MCP serversAI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
unrendered-mcp@0.1.4 -
Atalaya AI crawler check
MCP serversWhich AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
-
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
ai-crawler-readiness
MCP serversCan AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
-
Enrichment Crawlers MCP by InsightfulPipe
MCP serversHosted Enrichment Crawlers MCP server, with each action switched on or off per account.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
NM Signals: AI crawler audit
MCP serversAudit webpage access for AI crawlers. API key and Premium or Partner plan required.
-
GenPark Bloom Filter URL Dedup Crawler
MCP serversIn-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb -
Site SEO & AI Crawler Audit
MCP serversCheck public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
Web Crawler MCP by InsightfulPipe
MCP serversHosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
-
CrawlerData: Google Shopping & SERP
MCP serversGoogle Shopping products and seller prices, plus Google Search, Images, News and Local results.
-
The Crawler Zoo Arena
MCP serversGames for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
-
AI Crawler Access Checker
MCP serversShows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.