InnerVault

Library

85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay

    Stale Official MCP Registry, io.github.kriton-dev Updated 14 days ago 1 note

  2. CrawlConsole

    MCP servers

    Backlink Analysis MCP for AI Agents: authority, referring domains, and competitor link gaps.

    Stale Official MCP Registry, com.crawlconsole Updated 6 months ago 2 notes

    @crawlconsole/mcp@0.1.4
  3. crawl-readiness

    MCP servers

    Check what AI crawlers may read and what a page permits, generate llms.txt and schema.

    Stale Official MCP Registry, io.github.Grubbomatic Updated 17 days ago 2 notes

    crawl-readiness-mcp@0.2.9
  4. polite-crawler

    MCP servers

    A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

    Indexed Official MCP Registry, io.github.mark1russell7 Updated yesterday 1 note

    polite-crawler-mcp@0.7.6
  5. crawlgraph-mcp

    MCP servers

    Backlink lookups and competitor gap analysis on Common Crawl's open web graph (4.4B links).

    Stale Official MCP Registry, io.github.pucilpet Updated 4 months ago 2 notes

    https://github.com/pucilpet/crawlgraph-mcp/releases/download/v0.2.2/crawlgraph-mcp.mcpb
  6. Eclipse Crawlway

    MCP servers

    Pay-per-crawl licensing receipts: sealed identity+terms+fingerprint binding. LOCAL/fixture only.

    Stale Official MCP Registry, io.github.CYAN-ECLIPSE-TECHNOLOGIES-LLC Updated 20 days ago

  7. crawlio-browser

    MCP servers

    Bridge a live Chrome browser to your agent: 150 tools for capture, extraction, and observation.

    Stale Official MCP Registry, app.crawlio Updated 3 months ago 1 note

    crawlio-browser@1.11.0
  8. CrawlHive

    MCP servers

    Memecoin safety scanner for Solana, BNB and Robinhood Chain with an onchain track record

    Stale Official MCP Registry, xyz.crawlhive Updated 32 days ago 1 note

  9. Crawl Census

    MCP servers

    Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

    Stale Official MCP Registry, io.github.taylorsmithgg Updated 2 months ago 1 note

  10. AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

    Stale Official MCP Registry, io.github.missiongrowth Updated 26 days ago 1 note

    unrendered-mcp@0.1.4
  11. Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.

    Stale Official MCP Registry, com.atalaya-ai Updated 13 days ago 1 note

  12. CrawlWitness

    MCP servers

    AI consent & licensing check for any domain, with signed on-chain receipts. Paid via x402.

    Stale Official MCP Registry, com.crawlwitness Updated 15 days ago 1 note

  13. ai-crawler-readiness

    MCP servers

    Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

    Stale Official MCP Registry, com.lumnika Updated 58 days ago 1 note

  14. Agent Crawl Readiness

    MCP servers

    Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

    Stale Official MCP Registry, io.github.sadri-dridi Updated 54 days ago 1 note

  15. Hosted Enrichment Crawlers MCP server, with each action switched on or off per account.

    Stale Official MCP Registry, com.insightfulpipe Updated 26 days ago 1 note

  16. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  17. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  18. crawlinx-mcp

    MCP servers

    Free technical-SEO audit MCP: crawl a site, run checks, return an LLM-ready shareable report.

    Stale Official MCP Registry, io.github.im-ff Updated 4 months ago 1 note

  19. Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

    Stale Official MCP Registry, io.github.newmediamadman Updated 54 days ago 1 note

  20. In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge

    Stale Official MCP Registry, io.github.alphaparkinc Updated 26 days ago 1 note

    https://github.com/Alpha-Park/genpark-bloom-filter-url-dedup-crawler-skill/releases/download/v1.0.1/genpark-bloom-filter-url-dedup-crawler-skill-v1.0.1.mcpb
  21. Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI

    Stale Official MCP Registry, com.mambabuilt Updated 5 days ago 2 notes

    @mambalabsdev/mcp-site-crawler-links-tables@1.0.2
  22. Crawlwise Data MCP

    MCP servers

    Per-URL tech-stack, feed and sitemap change events for AI agents, delivered as signed webhooks.

    Stale Official MCP Registry, com.crawlwisedata Updated 31 days ago 2 notes

    crawlwise-mcp@0.1.2
  23. CrawlKit Scrapers

    MCP servers

    58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted

    Stale Official MCP Registry, io.github.CrawlKit560 Updated 11 days ago 1 note

  24. Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.

    Stale Official MCP Registry, app.openkrill Updated 26 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.