Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
CrawlHive
MCP serversMemecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record
-
Crawl Census
MCP serversAsk before you fetch: will this domain serve your crawler, refuse it, or charge it?
-
Unrendered: AI Crawler Checker
MCP serversAI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
unrendered-mcp@0.1.4 -
Atalaya AI crawler check
MCP serversWhich AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
-
dingo
AI repositoriesDingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
git clone https://github.com/MigoXLab/dingo.git -
CrawlWitness
MCP serversAI consent & licensing check for any domain, with signed on-chain receipts. Paid via x402.
-
ai-crawler-readiness
MCP serversCan AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
-
Agent Crawl Readiness
MCP serversBounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
-
Enrichment Crawlers MCP by InsightfulPipe
MCP serversHosted Enrichment Crawlers MCP server, with each action switched on or off per account.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
crawlinx-mcp
MCP serversFree technical-SEO audit MCP: crawl a site, run checks, return an LLM-ready shareable report.
-
NM Signals: AI crawler audit
MCP serversAudit webpage access for AI crawlers. API key and Premium or Partner plan required.
-
GenPark Bloom Filter URL Dedup Crawler
MCP serversIn-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb -
mcp-site-crawler-links-tables
MCP serversCrawl a website for links, file download links, HTML tables as rows, and lists, with no AI
@mambalabsdev/mcp-site-crawler-links-tables@1.0.2 -
Crawlwise Data MCP
MCP serversPer-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.
crawlwise-mcp@0.1.2 -
CrawlKit Scrapers
MCP servers58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted
-
Site SEO & AI Crawler Audit
MCP serversCheck public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
Web Crawler MCP by InsightfulPipe
MCP serversHosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
-
Website to Markdown Crawler
MCP serversCrawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
-
CrawlerData: Google Shopping & SERP
MCP serversGoogle Shopping products and seller prices, plus Google Search, Images, News and Local results.
-
The Crawler Zoo Arena
MCP serversGames for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.