InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. webclaw

    MCP servers

    Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

    Stale GitHub, 0xMassi 2,380 ★AGPL-3.0Updated 33 days ago

    git clone https://github.com/0xMassi/webclaw.git
  2. graphlit-mcp-server

    MCP servers

    Model Context Protocol (MCP) Server for Graphlit Platform

    Stale GitHub, graphlit 378 ★MITUpdated 10 months ago

    git clone https://github.com/graphlit/graphlit-mcp-server.git
  3. doc-scraper

    MCP servers

    Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

    Indexed GitHub, Sriram-PR 109 ★Apache-2.0Updated 2 days ago

    git clone https://github.com/Sriram-PR/doc-scraper.git
  4. firecrawl-mcp-server

    MCP servers

    Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

    Indexed GitHub, firecrawl 7,836 ★MITUpdated today

    git clone https://github.com/firecrawl/firecrawl-mcp-server.git
  5. pagesift

    MCP servers

    Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.

    Indexed GitHub, YOLOW92 274 ★Apache-2.0Updated today

    git clone https://github.com/YOLOW92/pagesift.git
  6. fastCRW

    MCP servers

    Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.

    Stale Official MCP Registry, io.github.us Updated 4 months ago 1 note

  7. 🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

    Stale GitHub, Evil0ctal 21.1k ★Apache-2.0Updated 14 days ago

    git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git
  8. ai-crawler-policy

    MCP servers

    Cloudflare Workers MCP server: ai-crawler-policy

    Stale Official MCP Registry, io.github.lazymac2x Updated 6 months ago 1 note

  9. Open SEO Crawler

    MCP servers

    Self-hosted SEO crawler: run site audits and AI crawler checks, then read the issues and pages.

    Stale Official MCP Registry, io.github.puneetindersingh Updated 31 days ago 1 note

    https://github.com/puneetindersingh/open-seo-crawler/releases/download/mcp-v1.0.0/open-seo-crawler-mcp.mcpb@1.0.0
  10. polite-crawler

    MCP servers

    A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

    Indexed Official MCP Registry, io.github.mark1russell7 Updated yesterday 1 note

    polite-crawler-mcp@0.7.6
  11. seo-crawler-mcp

    MCP servers

    Crawl and analyse websites for SEO errors using Crawlee with SQLite storage

    Stale Official MCP Registry, io.github.houtini-ai Updated 9 months ago 1 note

    @houtini/seo-crawler-mcp@2.0.1
  12. AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

    Stale Official MCP Registry, io.github.missiongrowth Updated 26 days ago 1 note

    unrendered-mcp@0.1.4
  13. Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.

    Stale Official MCP Registry, com.atalaya-ai Updated 13 days ago 1 note

  14. ai-crawler-readiness

    MCP servers

    Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

    Stale Official MCP Registry, com.lumnika Updated 58 days ago 1 note

  15. Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  16. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  17. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  18. Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

    Stale Official MCP Registry, io.github.newmediamadman Updated 54 days ago 1 note

  19. In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge

    Stale Official MCP Registry, io.github.alphaparkinc Updated 26 days ago 1 note

    https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb
  20. Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.

    Stale Official MCP Registry, app.openkrill Updated 26 days ago 1 note

  21. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  22. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  23. Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  24. Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

    Stale Official MCP Registry, io.github.Nero-Engine Updated 15 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.