InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. moli

    Agents & frameworks

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    Indexed GitHub, lexmount 23.8k ★Apache-2.0Updated today

    git clone https://github.com/lexmount/moli.git
  2. Crawl4AI

    Public APIs

    An open-source crawler that turns websites into Markdown ready for an AI model.

    Hand-checked InnerVault (hand-checked), unclecode 87.3k ★Apache-2.0

    pip install -U crawl4ai
  3. ai-crawler-py

    Agents & frameworks

    Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

    Stale Archived GitHub, oxylabs 3,262 ★Updated 2 months ago 1 note

    git clone https://github.com/oxylabs/ai-crawler-py.git
  4. AnyCrawl

    AI repositories

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    Indexed GitHub, any4ai 3,528 ★MITUpdated 4 days ago

    git clone https://github.com/any4ai/AnyCrawl.git
  5. trafilatura

    AI repositories

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

    Indexed GitHub, adbar 7,128 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/adbar/trafilatura.git
  6. Firecrawl

    Public APIs

    Turns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.

    Hand-checked InnerVault (hand-checked), Firecrawl 201.9k ★AGPL-3.0

  7. webclaw

    MCP servers

    Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

    Stale GitHub, 0xMassi 2,380 ★AGPL-3.0Updated 33 days ago

    git clone https://github.com/0xMassi/webclaw.git
  8. firecrawl-mcp-server

    MCP servers

    Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

    Indexed GitHub, firecrawl 7,836 ★MITUpdated today

    git clone https://github.com/firecrawl/firecrawl-mcp-server.git
  9. Scrapegraph-ai

    AI repositories

    Python scraper based on AI

    Stale GitHub, ScrapeGraphAI 31.8k ★MITUpdated 41 days ago

    git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git
  10. searxNcrawl

    AI repositories

    Open-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.

    Stale GitHub, DasDigitaleMomentum 189 ★MITUpdated 5 days ago

    git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git
  11. fastCRW

    MCP servers

    Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.

    Stale Official MCP Registry, io.github.us Updated 4 months ago 1 note

  12. wigolo

    AI repositories

    The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

    Indexed GitHub, KnockOutEZ 5,546 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/KnockOutEZ/wigolo.git
  13. awesome-ai-web-scraping

    Docs & tutorials

    A curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.

    Stale GitHub, h4ckf0r0day 114 ★CC0-1.0Updated 6 months ago

    git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git
  14. olostep-mcp-server

    MCP servers

    MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.

    Stale GitHub, olostep 25 ★MITUpdated 6 days ago

    git clone https://github.com/olostep/olostep-mcp-server.git
  15. Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.

    Stale Official MCP Registry, dev.oassis Updated 25 days ago 1 note

  16. CRW Web Scraper

    MCP servers

    Open-source web scraper for AI agents with scrape, crawl, and map tools

    Indexed Official MCP Registry, io.github.fastcrw 1,259 ★Updated today 1 note

    ghcr.io/fastcrw/crw:latest
  17. doc-scraper

    MCP servers

    Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

    Indexed GitHub, Sriram-PR 109 ★Apache-2.0Updated 2 days ago

    git clone https://github.com/Sriram-PR/doc-scraper.git
  18. Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.

    Stale Official MCP Registry, io.github.ByteBlisk Updated 8 days ago 1 note

  19. Scrape-LE

    MCP servers

    Analyse robots.txt content and report whether a path may be crawled.

    Stale Official MCP Registry, io.github.nolindnaidoo Updated 20 days ago 1 note

    scrape-le-mcp@2.6.0
  20. GoAI Moat Web Scrape

    MCP servers

    Scrape any URL into clean LLM-ready Markdown, or crawl a whole site.

    Stale Official MCP Registry, com.goaimoat Updated 53 days ago 1 note

  21. Website crawling, Google search and Wikipedia lookup as agent tools.

    Stale Official MCP Registry, com.thenextgennexus Updated 9 days ago 1 note

  22. graphlit-mcp-server

    MCP servers

    Model Context Protocol (MCP) Server for Graphlit Platform

    Stale GitHub, graphlit 378 ★MITUpdated 10 months ago

    git clone https://github.com/graphlit/graphlit-mcp-server.git
  23. Apify MCP server

    MCP servers

    Lets an agent run Apify's ready-made scrapers to pull data from websites, search engines, maps and social platforms.

    Hand-checked InnerVault (hand-checked), Apify 16.0k ★MIT

  24. ProxyCrawl

    Public APIs

    Scraping and crawling anticaptcha service

    Indexed public-apis (GitHub list)

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.