Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
CrawlConsole
MCP serversBacklink Analysis MCP for AI Agents: authority, referring domains, and competitor link gaps.
@crawlconsole/mcp@0.1.4 -
crawl-readiness
MCP serversCheck what AI crawlers may read and what a page permits, generate llms.txt and schema.
crawl-readiness-mcp@0.2.9 -
polite-crawler
MCP serversA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
polite-crawler-mcp@0.7.6 -
crawlgraph-mcp
MCP serversBacklink lookups and competitor gap analysis on Common Crawl's open web graph (4.4B links).
https://github.com/pucilpet/crawlgraph-mcp/releases/download/v0.2.2/crawlgraph-mcp.mcpb -
Eclipse Crawlway
MCP serversPay-per-crawl licensing receipts: sealed identity+terms+fingerprint binding. LOCAL/fixture only.
-
crawlio-browser
MCP serversBridge a live Chrome browser to your agent: 150 tools for capture, extraction, and observation.
crawlio-browser@1.11.0 -
CrawlHive
MCP serversMemecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record
-
Crawl Census
MCP serversAsk before you fetch: will this domain serve your crawler, refuse it, or charge it?
-
Unrendered: AI Crawler Checker
MCP serversAI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
unrendered-mcp@0.1.4 -
Atalaya AI crawler check
MCP serversWhich AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
-
CrawlWitness
MCP serversAI consent & licensing check for any domain, with signed on-chain receipts. Paid via x402.
-
ai-crawler-readiness
MCP serversCan AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
-
Agent Crawl Readiness
MCP serversBounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
-
Enrichment Crawlers MCP by InsightfulPipe
MCP serversHosted Enrichment Crawlers MCP server, with each action switched on or off per account.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
crawlinx-mcp
MCP serversFree technical-SEO audit MCP: crawl a site, run checks, return an LLM-ready shareable report.
-
NM Signals: AI crawler audit
MCP serversAudit webpage access for AI crawlers. API key and Premium or Partner plan required.
-
GenPark Bloom Filter URL Dedup Crawler
MCP serversIn-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb -
mcp-site-crawler-links-tables
MCP serversCrawl a website for links, file download links, HTML tables as rows, and lists, with no AI
@mambalabsdev/mcp-site-crawler-links-tables@1.0.2 -
Crawlwise Data MCP
MCP serversPer-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.
crawlwise-mcp@0.1.2 -
CrawlKit Scrapers
MCP servers58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted
-
Site SEO & AI Crawler Audit
MCP serversCheck public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.