InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay

    Stale Official MCP Registry, io.github.kriton-dev Updated 14 days ago 1 note

  2. Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.

    Stale Official MCP Registry, io.github.ByteBlisk Updated 8 days ago 1 note

  3. fess

    AI repositories

    Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

    Indexed GitHub, codelibs 1,151 ★Apache-2.0Updated yesterday

    git clone https://github.com/codelibs/fess.git
  4. One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

    Stale Official MCP Registry, app.rowb.ai-visibility Updated 54 days ago 1 note

  5. 🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

    Stale GitHub, Evil0ctal 21.1k ★Apache-2.0Updated 14 days ago

    git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git
  6. Agent skills for fetching knowledge

    Stale GitHub, diffbot 5 ★MITUpdated 8 days ago

    git clone https://github.com/diffbot/diffbot-skills.git
  7. AnyCrawl

    AI repositories

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    Indexed GitHub, any4ai 3,528 ★MITUpdated 4 days ago

    git clone https://github.com/any4ai/AnyCrawl.git
  8. pagesift

    MCP servers

    Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.

    Indexed GitHub, YOLOW92 274 ★Apache-2.0Updated today

    git clone https://github.com/YOLOW92/pagesift.git
  9. seo-ops

    Skills

    SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.

    Stale GitHub, tigerless-labs 951 ★Updated 21 days ago

    git clone https://github.com/tigerless-labs/seo-ops.git
  10. IndexLinks

    MCP servers

    Check how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.

    Stale Official MCP Registry, app.indexlinks Updated 42 days ago 1 note

  11. Siteiz

    MCP servers

    Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.

    Stale Official MCP Registry, com.siteiz Updated 44 days ago 1 note

  12. UAParse

    MCP servers

    Parse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.

    Stale Official MCP Registry, dev.workers.mike-tusa.uaparse Updated 15 days ago 1 note

  13. skald

    MCP servers

    Self-hosted Google Analytics alternative that reads your server access log: real people, AI assistants, crawlers, bots and the scanners probing your site, which JavaScript analytics never see. Leads and revenue by source. Every figure counted twice. No cookies, no third-party script. Python, SQLite, Docker.

    Indexed GitHub, spcaeo 22 ★Custom licence (see repository)Updated 3 days ago

    git clone https://github.com/spcaeo/skald.git
  14. awesome-ai-web-scraping

    Docs & tutorials

    A curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.

    Stale GitHub, h4ckf0r0day 114 ★CC0-1.0Updated 6 months ago

    git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git
  15. basemind

    AI repositories

    Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

    Indexed GitHub, Goldziher 121 ★MITUpdated today

    git clone https://github.com/Goldziher/basemind.git
  16. crawlproof

    MCP servers

    AI-crawler audits + ads (crawlproof.com)

    Stale Official MCP Registry, com.profullstack Updated 5 days ago 1 note

  17. geo-skills

    Skills

    9 open Claude skills for GEO / AEO: check if AI search can read a page, collect and analyze how often AI engines name a brand (with confidence ranges), plus schema parity, AI crawler log checks and migration guards. From the AI Visibility Census.

    Indexed GitHub, aivisibilitycensus 5 ★MITUpdated 2 days ago

    git clone https://github.com/aivisibilitycensus/geo-skills.git
  18. RealBots

    Public APIs

    Verify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real

    Indexed public-apis (GitHub list)

  19. CitedBy

    Public APIs

    Which AI crawlers can read a site: robots.txt and server answer for each

    Indexed public-apis (GitHub list)

  20. visprobe

    MCP servers

    AI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.

    Stale Official MCP Registry, io.github.CloudCorpRecords Updated 14 days ago 1 note

    @reneturcios/visprobe@0.1.0
  21. Chad Analytics

    MCP servers

    Read-only website analytics, AI referrals, crawlers, goals, revenue, and funnels.

    Stale Official MCP Registry, com.chadanalytics Updated 3 months ago 1 note

  22. rattlesnakesbymail

    MCP servers

    Dated, sourced claims about AI crawlers and agents, offered as MCP tools.

    Stale Official MCP Registry, com.rattlesnakesbymail Updated 58 days ago 1 note

  23. geolint

    MCP servers

    ESLint for AI search: audits AI-crawler access, llms.txt, schema and citability.

    Stale Official MCP Registry, io.github.iliasabk Updated 2 months ago 1 note

    @iliasabk/geolint@0.4.0
  24. Crawlwise

    MCP servers

    SEO audits and AI crawler checks. Honest findings only; no invented scores.

    Stale Official MCP Registry, site.crawlwise Updated 11 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.