Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
ai-crawler-readiness
MCP serversCan AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
-
Enrichment Crawlers MCP by InsightfulPipe
MCP serversHosted Enrichment Crawlers MCP server, with each action switched on or off per account.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
ActableSite AI Crawler Monitor
MCP serversCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
NM Signals: AI crawler audit
MCP serversAudit webpage access for AI crawlers. API key and Premium or Partner plan required.
-
GenPark Bloom Filter URL Dedup Crawler
MCP serversIn-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge
https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb -
boss-zhipin-scraper
SkillsBoss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.
git clone https://github.com/eatmoreduck/boss-zhipin-scraper.git -
Site SEO & AI Crawler Audit
MCP serversCheck public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
-
robots-txt-ai-crawler-audit
MCP serversChecks robots.txt line by line and says what each AI crawler token actually controls - training, AI
@readystack/robots-txt-ai-crawler-audit@0.1.3 -
Robots.txt & AI Crawler Checker
MCP serversrobots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
-
Web Crawler MCP by InsightfulPipe
MCP serversHosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
-
AI Crawler Access Checker
MCP serversShows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
-
AI Crawler Access Check by Citant.ai
MCP serversCan ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.
-
Crimson Crawler
MCP serversEnrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.
-
The Crawler Zoo Arena
MCP serversGames for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
-
mcp-site-crawler-links-tables
MCP serversCrawl a website for links, file download links, HTML tables as rows, and lists, with no AI
@mambalabsdev/mcp-site-crawler-links-tables@1.0.2 -
smart-crawler
MCP serversE-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
Website to Markdown Crawler
MCP serversCrawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
-
Code Crawler
MCP serversStatic and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
@nithin554/code-crawler@0.1.92 -
Flarum Forum Scraper - Discussion Crawler Tool
MCP serversSearch or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…
-
Web Scraper & Crawler for RAG (Markdown)
MCP serversWebsite to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.