Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
CRW Web Scraper
MCP serversOpen-source web scraper for AI agents with scrape, crawl, and map tools
ghcr.io/fastcrw/crw:latest -
seo-crawler-mcp
MCP serversCrawl and analyse websites for SEO errors using Crawlee with SQLite storage
@houtini/seo-crawler-mcp@2.0.1 -
Nstdata Crawl
MCP serversScrape webpages, run crawls, and read Markdown and other results with Nstdata Crawl.
-
Crawlwise
MCP serversSEO audits and AI crawler checks. Honest findings only; no invented scores.
-
Crawlerpeek
MCP serversCheck whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
crawlerpeek@0.3.1 -
Crawlie
MCP serversTechnical SEO + GEO (AI-search) site audits: hosted crawls, prioritized fixes, report diffs.
-
Crawl4Agent
MCP serverscrawl4ai-compatible web crawler API: POST a URL, get clean LLM-ready Markdown with fit_markdown...
-
Crawlgraph
MCP serversCrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph (4.4B edges, 120M…
@pipeworx/mcp-crawlgraph@0.2.1 -
ai-crawler-policy
MCP serversCloudflare Workers MCP server: ai-crawler-policy
-
Open SEO Crawler
MCP serversSelf-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
https://github.com/puneetindersingh/open-seo-crawler/releases/download/mcp-v1.0.0/open-seo-crawler-mcp.mcpb@1.0.0 -
crawlport
MCP serversFirecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.
@reneturcios/crawlport@0.1.0 -
Crawlberg
MCP serversScrape, crawl, and map websites to Markdown or JSON via local CLI.
ghcr.io/xberg-io/crawlberg:1.12.3 -
CrawlRaven
MCP serversRead Search Console performance, keyword opportunities and annotations for your sites.
-
crawlforge-mcp-server
MCP serversWeb scraping, crawling, deep research & autonomous extraction — 31 MCP tools, clean Markdown/JSON
crawlforge-mcp-server@6.25.1 -
Crawlora MCP
MCP serversHosted MCP: 4684 structured web-data tools across 534 platform groups.
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
Crawl Readiness
Public APIsCheck what 50+ AI crawlers can read on a website and whether a page permits an AI use
-
Ultra-FineWeb-L1
Datasets & benchmarksHugging Face dataset by openbmb
openbmb/Ultra-FineWeb-L1 -
CrawlConsole
MCP serversBacklink Analysis MCP for AI Agents: authority, referring domains, and competitor link gaps.
@crawlconsole/mcp@0.1.4 -
crawl-readiness
MCP serversCheck what AI crawlers may read and what a page permits, generate llms.txt and schema.
crawl-readiness-mcp@0.2.9 -
polite-crawler
MCP serversA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
polite-crawler-mcp@0.7.6 -
crawlgraph-mcp
MCP serversBacklink lookups and competitor gap analysis on Common Crawl's open web graph (4.4B links).
https://github.com/pucilpet/crawlgraph-mcp/releases/download/v0.2.2/crawlgraph-mcp.mcpb -
Eclipse Crawlway
MCP serversPay-per-crawl licensing receipts: sealed identity+terms+fingerprint binding. LOCAL/fixture only.
-
crawlio-browser
MCP serversBridge a live Chrome browser to your agent: 150 tools for capture, extraction, and observation.
crawlio-browser@1.11.0
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.