Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
AI Crawler Access Checker
MCP serversShows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
-
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
-
browserpaw
MCP serversBrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
git clone https://github.com/GoldenLoaf24h/browserpaw.git -
olostep-mcp-server
MCP serversMCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
git clone https://github.com/olostep/olostep-mcp-server.git -
AI Crawler Access Check by Citant.ai
MCP serversCan ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.
-
Web Scraper & Crawler for RAG (Markdown)
MCP serversWebsite to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
-
Crimson Crawler
MCP serversEnrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.
-
basemind
AI repositoriesFull AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
git clone https://github.com/Goldziher/basemind.git -
facebook-ads-library-mcp
MCP serversFree, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research — advertiser, copy, landing pages, CTAs, any country.
git clone https://github.com/RamsesAguirre777/facebook-ads-library-mcp.git -
Find broken links on a website: crawls its pages and reports 404s, errors and redirect chains
-
smart-crawler
MCP serversE-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.
-
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
Code Crawler
MCP serversStatic and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
@nithin554/code-crawler@0.1.92 -
CrawlCheck
MCP serversVerification layer for the agentic web: a signed answer about any site before an agent acts.
-
CrawlEyes MCP Server
MCP serversWeb scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.
crawleyes@0.2.0 -
Flarum Forum Scraper - Discussion Crawler Tool
MCP serversSearch or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…
-
crawlbit
MCP serversFour free tools to check if AI engines can find, read and recognise a site. No API key.
crawlbit-mcp@1.0.1 -
AI Crawler Index
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Crawler Log Triage
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Douyin_TikTok_Download_API
MCP servers🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git -
Turn a website into Markdown with headings, links and word counts.
-
IndexLinks
MCP serversCheck how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.