Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Scrapegraph-ai
AI repositoriesPython scraper based on AI
git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git -
moli
Agents & frameworksBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
git clone https://github.com/lexmount/moli.git -
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git -
Web Crawler MCP by InsightfulPipe
MCP serversHosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
-
ai-crawler-py
Agents & frameworksCrawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
git clone https://github.com/oxylabs/ai-crawler-py.git -
graphlit-mcp-server
MCP serversModel Context Protocol (MCP) Server for Graphlit Platform
git clone https://github.com/graphlit/graphlit-mcp-server.git -
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
wigolo
AI repositoriesThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
git clone https://github.com/KnockOutEZ/wigolo.git -
webclaw
MCP serversFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
git clone https://github.com/0xMassi/webclaw.git -
pagesift
MCP serversLocal-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
git clone https://github.com/YOLOW92/pagesift.git -
searxNcrawl
AI repositoriesOpen-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.
git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git -
basemind
AI repositoriesFull AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
git clone https://github.com/Goldziher/basemind.git -
Brainiall Web
MCP serversWeb search, URL content extraction to Markdown, site mapping, and recursive web crawler.
-
acrawl
MCP serversAutonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.
https://github.com/Mingye-Lu/AgenticCrawler/releases/download/v0.17.0/acrawl-mcp-linux-x64.mcpb -
Crawl4Agent
MCP serverscrawl4ai-compatible web crawler API: POST a URL, get clean LLM-ready Markdown with fit_markdown...
-
GenPark Bloom Filter URL Dedup Crawler
MCP serversIn-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb -
lilbee
MCP serversA local model engine with a search engine and web crawler built in. Run your coding agents on your own GPUs, and get cited answers from your files, code, and the sites you crawl. One executable.
git clone https://github.com/tobocop2/lilbee.git
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.