Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawler IP Verifier - real Googlebot or fake
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
-
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
-
seo-audit-console
MCP serversTechnical SEO in Claude: GSC + crawl + DataForSEO - audit, portfolio, market sizing, content recon.
@houtini/seo-audit-console@1.0.5 -
seo-ops
SkillsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
git clone https://github.com/tigerless-labs/seo-ops.git -
MeshArc
MCP serversCrawl and scrape websites to clean markdown, and track what changed between crawls.
-
diffbot-skills
SkillsAgent skills for fetching knowledge
git clone https://github.com/diffbot/diffbot-skills.git -
Black SEO Analyzer
MCP serversAudit a page or crawl a whole site for technical SEO issues, locally. Compares crawls.
https://github.com/sethblack/black-seo-analyzer/releases/download/v26.11.3/bseoa-26.11.3-win32.mcpb@26.11.3 -
fastCRW
MCP serversScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
-
MMClaw
Agents & frameworksUltra-Lightweight, Pure Python Multimodal Agent.
git clone https://github.com/CrawlScript/MMClaw.git -
Siteiz
MCP serversCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
-
UAParse
MCP serversParse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.
-
Commoncrawl
MCP serversCommon Crawl's public web archive — find every time a URL was crawled since 2008 and read the…
@pipeworx/mcp-commoncrawl@0.2.1 -
wet
MCP serversOpen-source MCP server for AI agents: web search, content extraction, and library docs -- 5-strategy scraping, runs without API keys.
git clone https://github.com/n24q02m/wet.git -
skald
MCP serversSelf-hosted Google Analytics alternative that reads your server access log: real people, AI assistants, crawlers, bots and the scanners probing your site, which JavaScript analytics never see. Leads and revenue by source. Every figure counted twice. No cookies, no third-party script. Python, SQLite, Docker.
git clone https://github.com/spcaeo/skald.git -
ProxyCrawl
Public APIsScraping and crawling anticaptcha service
-
cli
SkillsFirecrawl CLI and agent skills: search, scrape, and interact with the web from your terminal or any coding agent, plus parse, crawl, map, Agent and Monitor. Firecrawl is the context API for AI agents.
git clone https://github.com/firecrawl/cli.git -
lilbee
MCP serversA local model engine with a search engine and web crawler built in. Run your coding agents on your own GPUs, and get cited answers from your files, code, and the sites you crawl. One executable.
git clone https://github.com/tobocop2/lilbee.git -
awesome-ai-web-scraping
Docs & tutorialsA curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.
git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git -
Svipall
MCP serversRead any page past its anti-bot wall, crawl sites, search without an API key, solve captchas locally
svipall@2.1.0 -
skills
SkillsFirecrawl agent skills for Claude Code, Codex and Cursor: web search, scraping, crawling and browser interaction. Firecrawl is the most complete web data API for AI agents.
git clone https://github.com/firecrawl/skills.git -
geo-skills
Skills9 open Claude skills for GEO / AEO: check if AI search can read a page, collect and analyze how often AI engines name a brand (with confidence ranges), plus schema parity, AI crawler log checks and migration guards. From the AI Visibility Census.
git clone https://github.com/aivisibilitycensus/geo-skills.git -
Webclaw
Public APIsWeb content extraction for LLMs with scrape, crawl, search, and summarize
-
firecrawl
MCP serversScrape, crawl, and extract structured data from websites at scale
mcparmory-firecrawl@1.0.3
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.