Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
crawlport
MCP serversFirecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.
@reneturcios/crawlport@0.1.0 -
basemind
AI repositoriesFull AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
git clone https://github.com/Goldziher/basemind.git -
CRW Web Scraper
MCP serversOpen-source web scraper for AI agents with scrape, crawl, and map tools
ghcr.io/fastcrw/crw:latest -
Crawlberg
MCP serversScrape, crawl, and map websites to Markdown or JSON via local CLI.
ghcr.io/xberg-io/crawlberg:1.12.3 -
Nstdata Crawl
MCP serversScrape webpages, run crawls, and read Markdown and other results with Nstdata Crawl.
-
skald
MCP serversSelf-hosted Google Analytics alternative that reads your server access log: real people, AI assistants, crawlers, bots and the scanners probing your site, which JavaScript analytics never see. Leads and revenue by source. Every figure counted twice. No cookies, no third-party script. Python, SQLite, Docker.
git clone https://github.com/spcaeo/skald.git -
crawlforge-mcp-server
MCP serversWeb scraping, crawling, deep research & autonomous extraction — 31 MCP tools, clean Markdown/JSON
crawlforge-mcp-server@6.25.1 -
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
Scrape-LE
MCP serversAnalyse robots.txt content and report whether a path may be crawled.
scrape-le-mcp@2.6.0 -
RealBots
Public APIsVerify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real
-
web-scraping-mcp-server
MCP serversWebsite crawling, Google search and Wikipedia lookup as agent tools.
-
GoAI Moat Web Scrape
MCP serversScrape any URL into clean LLM-ready Markdown, or crawl a whole site.
-
wet
MCP serversOpen-source MCP server for AI agents: web search, content extraction, and library docs -- 5-strategy scraping, runs without API keys.
git clone https://github.com/n24q02m/wet.git -
cli
SkillsFirecrawl CLI and agent skills: search, scrape, and interact with the web from your terminal or any coding agent, plus parse, crawl, map, Agent and Monitor. Firecrawl is the context API for AI agents.
git clone https://github.com/firecrawl/cli.git -
ProxyCrawl
Public APIsScraping and crawling anticaptcha service
-
CitedBy
Public APIsWhich AI crawlers can read a site: robots.txt and server answer for each
-
skills
SkillsFirecrawl agent skills for Claude Code, Codex and Cursor: web search, scraping, crawling and browser interaction. Firecrawl is the most complete web data API for AI agents.
git clone https://github.com/firecrawl/skills.git -
geo-skills
Skills9 open Claude skills for GEO / AEO: check if AI search can read a page, collect and analyze how often AI engines name a brand (with confidence ranges), plus schema parity, AI crawler log checks and migration guards. From the AI Visibility Census.
git clone https://github.com/aivisibilitycensus/geo-skills.git -
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
visprobe
MCP serversAI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.
@reneturcios/visprobe@0.1.0 -
seo-ops
SkillsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
git clone https://github.com/tigerless-labs/seo-ops.git -
CrawlEyes MCP Server
MCP serversWeb scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.
crawleyes@0.2.0 -
Chad Analytics
MCP serversRead-only website analytics, AI referrals, crawlers, goals, revenue, and funnels.
-
rattlesnakesbymail
MCP serversDated, sourced claims about AI crawlers and agents, offered as MCP tools.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.