Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl4AI
Public APIsAn open-source crawler that turns websites into Markdown ready for an AI model.
pip install -U crawl4ai -
Scrapegraph-ai
AI repositoriesPython scraper based on AI
git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git -
trafilatura
AI repositoriesPython & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
git clone https://github.com/adbar/trafilatura.git -
ai-crawler-py
Agents & frameworksCrawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
git clone https://github.com/oxylabs/ai-crawler-py.git -
moli
Agents & frameworksBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
git clone https://github.com/lexmount/moli.git -
webclaw
MCP serversFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
git clone https://github.com/0xMassi/webclaw.git -
AnyCrawl
AI repositoriesAnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
git clone https://github.com/any4ai/AnyCrawl.git -
searxNcrawl
AI repositoriesOpen-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.
git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git -
Firecrawl
Public APIsTurns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.
-
graphlit-mcp-server
MCP serversModel Context Protocol (MCP) Server for Graphlit Platform
git clone https://github.com/graphlit/graphlit-mcp-server.git -
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git -
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
awesome-ai-web-scraping
Docs & tutorialsA curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.
git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git -
wigolo
AI repositoriesThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
git clone https://github.com/KnockOutEZ/wigolo.git -
pagesift
MCP serversLocal-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
git clone https://github.com/YOLOW92/pagesift.git -
fastCRW
MCP serversScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
-
Douyin_TikTok_Download_API
MCP servers🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git -
xiaobei
Agents & frameworks为OPC/中小微企业量身打造的自媒体获客智能体
git clone https://github.com/TeamWiseFlow/xiaobei.git -
ai-crawler-policy
MCP serversCloudflare Workers MCP server: ai-crawler-policy
-
Open SEO Crawler
MCP serversSelf-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.
https://github.com/puneetindersingh/open-seo-crawler/releases/download/mcp-v1.0.0/open-seo-crawler-mcp.mcpb@1.0.0 -
polite-crawler
MCP serversA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
polite-crawler-mcp@0.7.6 -
seo-crawler-mcp
MCP serversCrawl and analyse websites for SEO errors using Crawlee with SQLite storage
@houtini/seo-crawler-mcp@2.0.1 -
Unrendered: AI Crawler Checker
MCP serversAI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.
unrendered-mcp@0.1.4 -
Atalaya AI crawler check
MCP serversWhich AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.