Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
Web Scraper & Crawler for RAG (Markdown)
MCP serversWebsite to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
-
fess
AI repositoriesOpen-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
git clone https://github.com/codelibs/fess.git -
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
-
Douyin_TikTok_Download_API
MCP servers🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git -
diffbot-skills
SkillsAgent skills for fetching knowledge
git clone https://github.com/diffbot/diffbot-skills.git -
AnyCrawl
AI repositoriesAnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
git clone https://github.com/any4ai/AnyCrawl.git -
pagesift
MCP serversLocal-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
git clone https://github.com/YOLOW92/pagesift.git -
seo-ops
SkillsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
git clone https://github.com/tigerless-labs/seo-ops.git -
IndexLinks
MCP serversCheck how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.
-
Siteiz
MCP serversCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
-
UAParse
MCP serversParse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.
-
skald
MCP serversSelf-hosted Google Analytics alternative that reads your server access log: real people, AI assistants, crawlers, bots and the scanners probing your site, which JavaScript analytics never see. Leads and revenue by source. Every figure counted twice. No cookies, no third-party script. Python, SQLite, Docker.
git clone https://github.com/spcaeo/skald.git -
awesome-ai-web-scraping
Docs & tutorialsA curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.
git clone https://github.com/h4ckf0r0day/awesome-ai-web-scraping.git -
basemind
AI repositoriesFull AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
git clone https://github.com/Goldziher/basemind.git -
crawlproof
MCP serversAI-crawler audits + ads (crawlproof.com)
-
geo-skills
Skills9 open Claude skills for GEO / AEO: check if AI search can read a page, collect and analyze how often AI engines name a brand (with confidence ranges), plus schema parity, AI crawler log checks and migration guards. From the AI Visibility Census.
git clone https://github.com/aivisibilitycensus/geo-skills.git -
RealBots
Public APIsVerify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real
-
CitedBy
Public APIsWhich AI crawlers can read a site: robots.txt and server answer for each
-
visprobe
MCP serversAI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.
@reneturcios/visprobe@0.1.0 -
Chad Analytics
MCP serversRead-only website analytics, AI referrals, crawlers, goals, revenue, and funnels.
-
rattlesnakesbymail
MCP serversDated, sourced claims about AI crawlers and agents, offered as MCP tools.
-
geolint
MCP serversESLint for AI search: audits AI-crawler access, llms.txt, schema and citability.
@iliasabk/geolint@0.4.0 -
Crawlwise
MCP serversSEO audits and AI crawler checks. Honest findings only; no invented scores.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.