InnerVault

Library

85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. matchstick

    MCP servers

    Checks a site build: sitemaps, structured data, robots.txt and what crawlers see.

    Stale Official MCP Registry, io.github.Astralchemist Updated 24 days ago 1 note

    @astralchemist/matchstick@0.1.0-alpha.4
  2. Crawlerpeek

    MCP servers

    Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.

    Stale Official MCP Registry, io.github.crawlerpeek Updated 19 days ago 1 note

    crawlerpeek@0.3.1
  3. Wholetally

    MCP servers

    Cookieless analytics for agents: visitors, pages, sources, AI crawlers, funnels and revenue.

    Stale Official MCP Registry, com.wholetally Updated 7 days ago 1 note

  4. MTScraper

    MCP servers

    Free local crawler: any site, JavaScript pages and linked PDFs, into clean text.

    Indexed Official MCP Registry, io.github.mazprotech Updated 2 days ago 1 note

    mtscraper@2.1.1
  5. Prerender Buddy

    MCP servers

    Crawler diagnostics, AI visibility evidence and article drafts for review in Prerender Buddy.

    Stale Official MCP Registry, io.github.kopachlager Updated 4 days ago 1 note

    @prerenderbuddy/mcp@0.4.0
  6. TidyTools

    Public APIs

    Web page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)

    Indexed public-apis (GitHub list)

  7. domaintrack

    MCP servers

    Look up domain reputation snapshots and AI crawler policy from Domduck's public store.

    Stale Official MCP Registry, io.github.CloudCorpRecords Updated 13 days ago 1 note

    @reneturcios/domaintrack@0.1.0
  8. AISeeWhy

    MCP servers

    Can ChatGPT, Claude and Perplexity crawlers read your page? Audit, fixes, compare, monitoring.

    Stale Official MCP Registry, com.aiseewhy Updated 5 days ago 1 note

  9. Brainiall Web

    MCP servers

    Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.

    Stale Official MCP Registry, com.brainiall Updated 2 months ago 1 note

  10. OpenAEO

    MCP servers

    Asks 8 AI crawlers for your page, checks who really answered, and writes the fixes. MIT.

    Stale Official MCP Registry, dev.openaeo Updated 3 months ago 2 notes

    openaeo-audit@1.6.4
  11. IndexPulse

    MCP servers

    Bing Webmaster, IndexNow and Google Search Console for AI: rankings, indexing gaps, AI crawlers.

    Stale Official MCP Registry, in.puzaricloud Updated 35 days ago 1 note

  12. acrawl

    MCP servers

    Autonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.

    Stale Official MCP Registry, io.github.Mingye-Lu Updated 20 days ago 1 note

    https://github.com/Mingye-Lu/AgenticCrawler/releases/download/v0.17.0/acrawl-mcp-linux-x64.mcpb
  13. aeo-mcp

    MCP servers

    AEO tools for AI agents: crawler permissions, llms.txt, structured data, AI-readiness audit.

    Stale Official MCP Registry, io.github.arthurcarlsonn Updated 4 months ago 1 note

    aeo-mcp@1.0.1
  14. pagerender

    MCP servers

    What crawlers see from any URL, plus page lookups and content editing for your Pagerender sites.

    Stale Official MCP Registry, io.pagerender Updated 8 days ago 2 notes

    @pagerender/mcp@1.3.0
  15. alice

    MCP servers

    Audit any site's AI visibility from your assistant: crawler access, rendering, and schema.

    Stale Official MCP Registry, io.tranx Updated 55 days ago 1 note

  16. web-shooter

    MCP servers

    Fast, low-token web reader and focused crawler. Returns ranked passages, not whole pages.

    Stale Official MCP Registry, io.github.heyyviv Updated 14 days ago 1 note

    web-shooter@0.2.0
  17. Scraperapi

    MCP servers

    ScraperAPI MCP — wraps ScraperAPI (scraperapi.com), a proxy-based web scraper with structured-data…

    Stale Official MCP Registry, io.github.pipeworx-io Updated 12 days ago 3 notes

    @pipeworx/mcp-scraperapi@0.2.1
  18. ScraperApi

    Public APIs

    Easily build scalable web scrapers

    Indexed public-apis (GitHub list)

  19. AI agent skills for social media research. Outlier posts, comment mining, competitor teardowns, ad libraries & trends across TikTok, Instagram, YouTube, Reddit, X, LinkedIn & more. Powered by ScrapeCreators. Works with Claude Code, Cursor, Codex, Gemini CLI.

    Stale GitHub, ScrapeCreators 3,688 ★MITUpdated 2 months ago

    git clone https://github.com/ScrapeCreators/social-media-research-skills.git
  20. Domduck

    MCP servers

    Domain reputation over time: Domain Rating, Tranco rank, TLS, and AI crawler policy.

    Stale Official MCP Registry, com.domduck Updated 3 months ago 1 note

  21. Crawl Readiness

    Public APIs

    Check what 50+ AI crawlers can read on a website and whether a page permits an AI use

    Indexed public-apis (GitHub list)

  22. Causabi GEO Optimizer

    MCP servers

    Audit a site for AI readiness: AI-crawler access, schema, FAQ, content structure. Generates fixes.

    Stale Official MCP Registry, com.causabi Updated 27 days ago 1 note

    geo-optimizer-mcp@0.5.1
  23. Botcard

    MCP servers

    Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.

    Stale Official MCP Registry, com.whosthatbot Updated 29 days ago 1 note

  24. useragent-mcp

    MCP servers

    Parse user-agent strings into browser, engine, OS and device, and flag crawlers - fully local.

    Stale Official MCP Registry, io.github.mrfentmen Updated 9 days ago 1 note

    @mrfentmen/useragent-mcp@1.0.1

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.