Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl4Agent
MCP serverscrawl4ai-compatible web crawler API: POST a URL, get clean LLM-ready Markdown with fit_markdown...
-
olostep-mcp-server
MCP serversMCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
git clone https://github.com/olostep/olostep-mcp-server.git -
crawl-readiness
MCP serversCheck what AI crawlers may read and what a page permits, generate llms.txt and schema.
crawl-readiness-mcp@0.2.9 -
Crawl Census
MCP serversAsk before you fetch: will this domain serve your crawler, refuse it, or charge it?
-
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
-
facebook-ads-library-mcp
MCP serversFree, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research — advertiser, copy, landing pages, CTAs, any country.
git clone https://github.com/RamsesAguirre777/facebook-ads-library-mcp.git -
Apify MCP server
MCP serversLets an agent run Apify's ready-made scrapers to pull data from websites, search engines, maps and social platforms.
-
crawlport
MCP serversFirecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.
@reneturcios/crawlport@0.1.0 -
CRW Web Scraper
MCP serversOpen-source web scraper for AI agents with scrape, crawl, and map tools
ghcr.io/fastcrw/crw:latest -
Crawlberg
MCP serversScrape, crawl, and map websites to Markdown or JSON via local CLI.
ghcr.io/xberg-io/crawlberg:1.12.3 -
Nstdata Crawl
MCP serversScrape webpages, run crawls, and read Markdown and other results with Nstdata Crawl.
-
skald
MCP serversSelf-hosted Google Analytics alternative that reads your server access log: real people, AI assistants, crawlers, bots and the scanners probing your site, which JavaScript analytics never see. Leads and revenue by source. Every figure counted twice. No cookies, no third-party script. Python, SQLite, Docker.
git clone https://github.com/spcaeo/skald.git -
crawlforge-mcp-server
MCP serversWeb scraping, crawling, deep research & autonomous extraction — 31 MCP tools, clean Markdown/JSON
crawlforge-mcp-server@6.25.1 -
Scrape-LE
MCP serversAnalyse robots.txt content and report whether a path may be crawled.
scrape-le-mcp@2.6.0 -
web-scraping-mcp-server
MCP serversWebsite crawling, Google search and Wikipedia lookup as agent tools.
-
GoAI Moat Web Scrape
MCP serversScrape any URL into clean LLM-ready Markdown, or crawl a whole site.
-
wet
MCP serversOpen-source MCP server for AI agents: web search, content extraction, and library docs -- 5-strategy scraping, runs without API keys.
git clone https://github.com/n24q02m/wet.git -
visprobe
MCP serversAI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.
@reneturcios/visprobe@0.1.0 -
CrawlEyes MCP Server
MCP serversWeb scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.
crawleyes@0.2.0 -
Chad Analytics
MCP serversRead-only website analytics, AI referrals, crawlers, goals, revenue, and funnels.
-
rattlesnakesbymail
MCP serversDated, sourced claims about AI crawlers and agents, offered as MCP tools.
-
geolint
MCP serversESLint for AI search: audits AI-crawler access, llms.txt, schema and citability.
@iliasabk/geolint@0.4.0 -
matchstick
MCP serversChecks a site build: sitemaps, structured data, robots.txt and what crawlers see.
@astralchemist/matchstick@0.1.0-alpha.4 -
Wholetally
MCP serversCookieless analytics for agents: visitors, pages, sources, AI crawlers, funnels and revenue.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.