InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. ai-crawler-readiness

    MCP servers

    Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

    Stale Official MCP Registry, com.lumnika Updated 58 days ago 1 note

  2. Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  3. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  4. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  5. Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

    Stale Official MCP Registry, io.github.newmediamadman Updated 54 days ago 1 note

  6. In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge

    Stale Official MCP Registry, io.github.alphaparkinc Updated 26 days ago 1 note

    https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb
  7. Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

    Stale GitHub, eatmoreduck 1,699 ★MITUpdated 13 days ago

    git clone https://github.com/eatmoreduck/boss-zhipin-scraper.git
  8. Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.

    Stale Official MCP Registry, app.openkrill Updated 26 days ago 1 note

  9. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  10. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  11. Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  12. Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

    Stale Official MCP Registry, io.github.Nero-Engine Updated 15 days ago 1 note

  13. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  14. Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.

    Stale Official MCP Registry, io.github.stellalytics Updated 3 days ago 1 note

  15. Can ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.

    Stale Official MCP Registry, io.github.hamzawamiqhq Updated 4 days ago 1 note

  16. Crimson Crawler

    MCP servers

    Enrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.

    Stale Official MCP Registry, com.crimsoncrawler Updated 3 months ago 1 note

  17. The Crawler Zoo Arena

    MCP servers

    Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.

    Stale Official MCP Registry, com.crawlerzoo Updated 23 days ago 1 note

  18. Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI

    Stale Official MCP Registry, com.mambabuilt Updated 5 days ago 2 notes

    @mambalabsdev/mcp-site-crawler-links-tables@1.0.2
  19. smart-crawler

    MCP servers

    E-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.

    Stale Official MCP Registry, io.github.shulex-voc Updated 5 months ago 1 note

  20. Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay

    Stale Official MCP Registry, io.github.kriton-dev Updated 14 days ago 1 note

  21. Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.

    Stale Official MCP Registry, io.github.prometheus-maker Updated 10 days ago 1 note

  22. Code Crawler

    MCP servers

    Static and dynamic call tracing MCP server backed by a SQLite property graph of your codebase

    Indexed Official MCP Registry, io.github.nithin554 Updated yesterday 1 note

    @nithin554/code-crawler@0.1.92
  23. Search or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…

    Stale Official MCP Registry, com.ranovam Updated 49 days ago 1 note

  24. Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.

    Stale Official MCP Registry, io.github.ByteBlisk Updated 8 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.