Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
moli
Agents & frameworksBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
git clone https://github.com/lexmount/moli.git -
Crawl4AI
Public APIsAn open-source crawler that turns websites into Markdown ready for an AI model.
pip install -U crawl4ai -
ai-crawler-py
Agents & frameworksCrawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
git clone https://github.com/oxylabs/ai-crawler-py.git -
AnyCrawl
AI repositoriesAnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
git clone https://github.com/any4ai/AnyCrawl.git -
trafilatura
AI repositoriesPython & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
git clone https://github.com/adbar/trafilatura.git -
Firecrawl
Public APIsTurns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.
-
webclaw
MCP serversFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
git clone https://github.com/0xMassi/webclaw.git -
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
Scrapegraph-ai
AI repositoriesPython scraper based on AI
git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git -
searxNcrawl
AI repositoriesOpen-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.
git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git -
fastCRW
MCP serversScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
-
wigolo
AI repositoriesThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
git clone https://github.com/KnockOutEZ/wigolo.git -
awesome-ai-web-scraping
Docs & tutorialsA curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.
git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git -
olostep-mcp-server
MCP serversMCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
git clone https://github.com/olostep/olostep-mcp-server.git -
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
-
CRW Web Scraper
MCP serversOpen-source web scraper for AI agents with scrape, crawl, and map tools
ghcr.io/fastcrw/crw:latest -
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git -
Web Scraper & Crawler for RAG (Markdown)
MCP serversWebsite to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
-
Scrape-LE
MCP serversAnalyse robots.txt content and report whether a path may be crawled.
scrape-le-mcp@2.6.0 -
GoAI Moat Web Scrape
MCP serversScrape any URL into clean LLM-ready Markdown, or crawl a whole site.
-
web-scraping-mcp-server
MCP serversWebsite crawling, Google search and Wikipedia lookup as agent tools.
-
graphlit-mcp-server
MCP serversModel Context Protocol (MCP) Server for Graphlit Platform
git clone https://github.com/graphlit/graphlit-mcp-server.git -
Apify MCP server
MCP serversLets an agent run Apify's ready-made scrapers to pull data from websites, search engines, maps and social platforms.
-
ProxyCrawl
Public APIsScraping and crawling anticaptcha service
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.