Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl4AI
Public APIsAn open-source crawler that turns websites into Markdown ready for an AI model.
pip install -U crawl4ai -
Firecrawl
Public APIsTurns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.
-
Scrapegraph-ai
AI repositoriesPython scraper based on AI
git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git -
moli
Agents & frameworksBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
git clone https://github.com/lexmount/moli.git -
exa-mcp-server
MCP serversExa MCP for web search and web crawling!
git clone https://github.com/exa-labs/exa-mcp-server.git -
xiaobei
Agents & frameworks为OPC/中小微企业量身打造的自媒体获客智能体
git clone https://github.com/TeamWiseFlow/xiaobei.git -
trafilatura
AI repositoriesPython & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
git clone https://github.com/adbar/trafilatura.git -
webclaw
MCP serversFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
git clone https://github.com/0xMassi/webclaw.git -
AnyCrawl
AI repositoriesAnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
git clone https://github.com/any4ai/AnyCrawl.git -
donsetch
Agents & frameworksWeb fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
git clone https://github.com/dondai44423/donsetch.git -
searxNcrawl
AI repositoriesOpen-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.
git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git -
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
wigolo
AI repositoriesThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
git clone https://github.com/KnockOutEZ/wigolo.git -
boss-zhipin-scraper
SkillsBoss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.
git clone https://github.com/eatmoreduck/boss-zhipin-scraper.git -
Crawllama
AI repositoriesCrawlLama 🦙 is an local AI agent that answers questions via Ollama and integrates web- and RAG-based research.
git clone https://github.com/arn-c0de/Crawllama.git -
ai-crawler-py
Agents & frameworksCrawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
git clone https://github.com/oxylabs/ai-crawler-py.git -
Apify MCP Server
MCP serversExtract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡
-
fess
AI repositoriesOpen-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
git clone https://github.com/codelibs/fess.git -
pagesift
MCP serversLocal-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
git clone https://github.com/YOLOW92/pagesift.git -
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git -
Leaked-GPTs
Docs & tutorialsLeaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
git clone https://github.com/friuns2/Leaked-GPTs.git -
Crawlbase (Unofficial)
MCP serversUnofficial Crawlbase MCP — wraps the Crawlbase Crawling API (crawlbase.com, formerly ProxyCrawl).…
@pipeworx/mcp-crawlbase@0.2.2 -
crawlproof
MCP serversAI-crawler audits + ads (crawlproof.com)
-
graphlit-mcp-server
MCP serversModel Context Protocol (MCP) Server for Graphlit Platform
git clone https://github.com/graphlit/graphlit-mcp-server.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.