InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Crawl4AI

    Public APIs

    An open-source crawler that turns websites into Markdown ready for an AI model.

    Hand-checked InnerVault (hand-checked), unclecode 87.3k ★Apache-2.0

    pip install -U crawl4ai
  2. Scrapegraph-ai

    AI repositories

    Python scraper based on AI

    Stale GitHub, ScrapeGraphAI 31.8k ★MITUpdated 41 days ago

    git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git
  3. trafilatura

    AI repositories

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

    Indexed GitHub, adbar 7,128 ★Apache-2.0Updated 3 days ago

    git clone https://github.com/adbar/trafilatura.git
  4. awesome-ai-web-scraping

    Docs & tutorials

    A curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.

    Stale GitHub, h4ckf0r0day 114 ★CC0-1.0Updated 6 months ago

    git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git
  5. moli

    Agents & frameworks

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    Indexed GitHub, lexmount 23.8k ★Apache-2.0Updated today

    git clone https://github.com/lexmount/moli.git
  6. webclaw

    MCP servers

    Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

    Stale GitHub, 0xMassi 2,380 ★AGPL-3.0Updated 33 days ago

    git clone https://github.com/0xMassi/webclaw.git
  7. AnyCrawl

    AI repositories

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    Indexed GitHub, any4ai 3,528 ★MITUpdated 4 days ago

    git clone https://github.com/any4ai/AnyCrawl.git
  8. doc-scraper

    MCP servers

    Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

    Indexed GitHub, Sriram-PR 109 ★Apache-2.0Updated 2 days ago

    git clone https://github.com/Sriram-PR/doc-scraper.git
  9. ai-crawler-py

    Agents & frameworks

    Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

    Stale Archived GitHub, oxylabs 3,262 ★Updated 2 months ago 1 note

    git clone https://github.com/oxylabs/ai-crawler-py.git
  10. Firecrawl

    Public APIs

    Turns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.

    Hand-checked InnerVault (hand-checked), Firecrawl 201.9k ★AGPL-3.0

  11. graphlit-mcp-server

    MCP servers

    Model Context Protocol (MCP) Server for Graphlit Platform

    Stale GitHub, graphlit 378 ★MITUpdated 10 months ago

    git clone https://github.com/graphlit/graphlit-mcp-server.git
  12. firecrawl-mcp-server

    MCP servers

    Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

    Indexed GitHub, firecrawl 7,836 ★MITUpdated today

    git clone https://github.com/firecrawl/firecrawl-mcp-server.git
  13. searxNcrawl

    AI repositories

    Open-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.

    Stale GitHub, DasDigitaleMomentum 189 ★MITUpdated 5 days ago

    git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git
  14. 🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

    Stale GitHub, Evil0ctal 21.1k ★Apache-2.0Updated 14 days ago

    git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git
  15. wigolo

    AI repositories

    The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

    Indexed GitHub, KnockOutEZ 5,546 ★Custom licence (see repository)Updated yesterday

    git clone https://github.com/KnockOutEZ/wigolo.git
  16. pagesift

    MCP servers

    Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.

    Indexed GitHub, YOLOW92 274 ★Apache-2.0Updated today

    git clone https://github.com/YOLOW92/pagesift.git
  17. fastCRW

    MCP servers

    Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.

    Stale Official MCP Registry, io.github.us Updated 4 months ago 1 note

  18. Playwright MCP

    MCP servers

    Lets an agent drive a real browser: open pages, click, fill in forms and read what's on screen.

    Hand-checked InnerVault (hand-checked), Microsoft Apache-2.0

    claude mcp add playwright npx @playwright/mcp@latest
  19. scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

    Indexed GitHub, gosom 8,121 ★MITUpdated yesterday

    git clone https://github.com/gosom/google-maps-scraper.git
  20. Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

    Stale GitHub, eatmoreduck 1,699 ★MITUpdated 13 days ago

    git clone https://github.com/eatmoreduck/boss-zhipin-scraper.git
  21. ketch

    Agents & frameworks

    Fast, stateless CLI for web search and scrape. Built for AI agents.

    Indexed GitHub, 1broseidon 748 ★MITUpdated 3 days ago

    git clone https://github.com/1broseidon/ketch.git
  22. nopecha-extension

    Agents & frameworks

    Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.

    Stale GitHub, NopeCHALLC 11.0k ★MITUpdated 45 days ago

    git clone https://github.com/NopeCHALLC/nopecha-extension.git
  23. decodo-openclaw-skill

    AI repositories

    OpenClaw skill for scraping any URL using the Decodo Web Scraping API.

    Stale GitHub, Decodo 152 ★Updated 18 days ago

    git clone https://github.com/Decodo/decodo-openclaw-skill.git
  24. llm-reader

    Agents & frameworks

    Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.

    Indexed GitHub, m92vyas 316 ★MITUpdated 6 days ago

    git clone https://github.com/m92vyas/llm-reader.git

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.