InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. CrawlHive

    MCP servers

    Memecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record

    Stale Official MCP Registry, xyz.crawlhive Updated 32 days ago 1 note

  2. Crawl Census

    MCP servers

    Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

    Stale Official MCP Registry, io.github.taylorsmithgg Updated 2 months ago 1 note

  3. AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

    Stale Official MCP Registry, io.github.missiongrowth Updated 26 days ago 1 note

    unrendered-mcp@0.1.4
  4. Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.

    Stale Official MCP Registry, com.atalaya-ai Updated 13 days ago 1 note

  5. dingo

    AI repositories

    Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

    Stale GitHub, MigoXLab 764 ★Apache-2.0Updated 13 days ago

    git clone https://github.com/MigoXLab/dingo.git
  6. CrawlWitness

    MCP servers

    AI consent & licensing check for any domain, with signed on-chain receipts. Paid via x402.

    Stale Official MCP Registry, com.crawlwitness Updated 15 days ago 1 note

  7. ai-crawler-readiness

    MCP servers

    Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

    Stale Official MCP Registry, com.lumnika Updated 58 days ago 1 note

  8. Agent Crawl Readiness

    MCP servers

    Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

    Stale Official MCP Registry, io.github.sadri-dridi Updated 54 days ago 1 note

  9. Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  10. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  11. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  12. crawlinx-mcp

    MCP servers

    Free technical-SEO audit MCP: crawl a site, run checks, return an LLM-ready shareable report.

    Stale Official MCP Registry, io.github.im-ff Updated 4 months ago 1 note

  13. Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

    Stale Official MCP Registry, io.github.newmediamadman Updated 54 days ago 1 note

  14. In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge

    Stale Official MCP Registry, io.github.alphaparkinc Updated 26 days ago 1 note

    https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb
  15. Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI

    Stale Official MCP Registry, com.mambabuilt Updated 5 days ago 2 notes

    @mambalabsdev/mcp-site-crawler-links-tables@1.0.2
  16. Crawlwise Data MCP

    MCP servers

    Per-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.

    Stale Official MCP Registry, com.crawlwisedata Updated 31 days ago 2 notes

    crawlwise-mcp@0.1.2
  17. CrawlKit Scrapers

    MCP servers

    58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted

    Stale Official MCP Registry, io.github.CrawlKit560 Updated 11 days ago 1 note

  18. Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.

    Stale Official MCP Registry, app.openkrill Updated 26 days ago 1 note

  19. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  20. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  21. Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  22. Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.

    Stale Official MCP Registry, io.github.prometheus-maker Updated 10 days ago 1 note

  23. Google Shopping products and seller prices, plus Google Search, Images, News and Local results.

    Stale Official MCP Registry, com.crawlerdata Updated 35 days ago 1 note

  24. The Crawler Zoo Arena

    MCP servers

    Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.

    Stale Official MCP Registry, com.crawlerzoo Updated 23 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.