InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Crawl4AI

    Public APIs

    An open-source crawler that turns websites into Markdown ready for an AI model.

    Hand-checked InnerVault (hand-checked), unclecode 87.3k ★Apache-2.0

    pip install -U crawl4ai
  2. Firecrawl

    Public APIs

    Turns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.

    Hand-checked InnerVault (hand-checked), Firecrawl 201.9k ★AGPL-3.0

  3. Scrapegraph-ai

    AI repositories

    Python scraper based on AI

    Stale GitHub, ScrapeGraphAI 31.8k ★MITUpdated 41 days ago

    git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git
  4. moli

    Agents & frameworks

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    Indexed GitHub, lexmount 23.8k ★Apache-2.0Updated today

    git clone https://github.com/lexmount/moli.git
  5. exa-mcp-server

    MCP servers

    Exa MCP for web search and web crawling!

    Stale GitHub, exa-labs 5,215 ★MITUpdated 11 days ago

    git clone https://github.com/exa-labs/exa-mcp-server.git
  6. xiaobei

    Agents & frameworks

    为OPC/中小微企业量身打造的自媒体获客智能体

    Stale GitHub, TeamWiseFlow 8,711 ★Custom licence (see repository)Updated 5 days ago

    git clone https://github.com/TeamWiseFlow/xiaobei.git
  7. trafilatura

    AI repositories

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

    Indexed GitHub, adbar 7,128 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/adbar/trafilatura.git
  8. webclaw

    MCP servers

    Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

    Stale GitHub, 0xMassi 2,380 ★AGPL-3.0Updated 33 days ago

    git clone https://github.com/0xMassi/webclaw.git
  9. AnyCrawl

    AI repositories

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    Indexed GitHub, any4ai 3,528 ★MITUpdated 4 days ago

    git clone https://github.com/any4ai/AnyCrawl.git
  10. donsetch

    Agents & frameworks

    Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

    Indexed GitHub, dondai44423 1,000 ★AGPL-3.0Updated today

    git clone https://github.com/dondai44423/donsetch.git
  11. searxNcrawl

    AI repositories

    Open-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.

    Stale GitHub, DasDigitaleMomentum 189 ★MITUpdated 5 days ago

    git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git
  12. firecrawl-mcp-server

    MCP servers

    Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

    Indexed GitHub, firecrawl 7,836 ★MITUpdated today

    git clone https://github.com/firecrawl/firecrawl-mcp-server.git
  13. wigolo

    AI repositories

    The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

    Indexed GitHub, KnockOutEZ 5,546 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/KnockOutEZ/wigolo.git
  14. Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

    Stale GitHub, eatmoreduck 1,699 ★MITUpdated 13 days ago

    git clone https://github.com/eatmoreduck/boss-zhipin-scraper.git
  15. Crawllama

    AI repositories

    CrawlLama 🦙 is an local AI agent that answers questions via Ollama and integrates web- and RAG-based research.

    Indexed GitHub, arn-c0de 109 ★Custom licence (see repository)Updated 3 days ago

    git clone https://github.com/arn-c0de/Crawllama.git
  16. ai-crawler-py

    Agents & frameworks

    Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

    Stale Archived GitHub, oxylabs 3,262 ★Updated 2 months ago 1 note

    git clone https://github.com/oxylabs/ai-crawler-py.git
  17. Apify MCP Server

    MCP servers

    Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡

    Indexed Official MCP Registry, com.apify 16.0k ★Updated today 1 note

  18. fess

    AI repositories

    Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

    Indexed GitHub, codelibs 1,151 ★Apache-2.0Updated yesterday

    git clone https://github.com/codelibs/fess.git
  19. pagesift

    MCP servers

    Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.

    Indexed GitHub, YOLOW92 274 ★Apache-2.0Updated today

    git clone https://github.com/YOLOW92/pagesift.git
  20. doc-scraper

    MCP servers

    Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

    Indexed GitHub, Sriram-PR 109 ★Apache-2.0Updated 2 days ago

    git clone https://github.com/Sriram-PR/doc-scraper.git
  21. Leaked-GPTs

    Docs & tutorials

    Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

    Stale GitHub, friuns2 2,475 ★Updated 2 years ago

    git clone https://github.com/friuns2/Leaked-GPTs.git
  22. Unofficial Crawlbase MCP — wraps the Crawlbase Crawling API (crawlbase.com, formerly ProxyCrawl).…

    Indexed Official MCP Registry, io.github.pipeworx-io Updated today 3 notes

    @pipeworx/mcp-crawlbase@0.2.2
  23. crawlproof

    MCP servers

    AI-crawler audits + ads (crawlproof.com)

    Stale Official MCP Registry, com.profullstack Updated 5 days ago 1 note

  24. graphlit-mcp-server

    MCP servers

    Model Context Protocol (MCP) Server for Graphlit Platform

    Stale GitHub, graphlit 378 ★MITUpdated 10 months ago

    git clone https://github.com/graphlit/graphlit-mcp-server.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.