InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  2. Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.

    Stale Official MCP Registry, io.github.stellalytics Updated 3 days ago 1 note

  3. Can ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.

    Stale Official MCP Registry, io.github.hamzawamiqhq Updated 4 days ago 1 note

  4. Crimson Crawler

    MCP servers

    Enrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.

    Stale Official MCP Registry, com.crimsoncrawler Updated 3 months ago 1 note

  5. Find broken links on a website: crawls its pages and reports 404s, errors and redirect chains

    Stale Official MCP Registry, io.github.kriton-dev Updated 14 days ago 1 note

  6. Ultra-FineWeb-L1

    Datasets & benchmarks

    Hugging Face dataset by openbmb

    Stale Hugging Face Hub, openbmb 201 ♥apache-2.0Updated 2 months ago

    openbmb/Ultra-FineWeb-L1
  7. smart-crawler

    MCP servers

    E-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.

    Stale Official MCP Registry, io.github.shulex-voc Updated 5 months ago 1 note

  8. Code Crawler

    MCP servers

    Static and dynamic call tracing MCP server backed by a SQLite property graph of your codebase

    Indexed Official MCP Registry, io.github.nithin554 Updated yesterday 1 note

    @nithin554/code-crawler@0.1.92
  9. dingo

    AI repositories

    Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

    Stale GitHub, MigoXLab 764 ★Apache-2.0Updated 13 days ago

    git clone https://github.com/MigoXLab/dingo.git
  10. browserpaw

    MCP servers

    BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus

    Stale GitHub, GoldenLoaf24h 38 ★AGPL-3.0Updated 14 days ago

    git clone https://github.com/GoldenLoaf24h/browserpaw.git
  11. The Crawler Zoo Arena

    MCP servers

    Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.

    Stale Official MCP Registry, com.crawlerzoo Updated 23 days ago 1 note

  12. AI Crawler Index

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  13. Crawler Log Triage

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  14. Turn a website into Markdown with headings, links and word counts.

    Stale Official MCP Registry, io.github.bepolabs Updated 4 days ago 1 note

  15. {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  16. One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

    Stale Official MCP Registry, app.rowb.ai-visibility Updated 54 days ago 1 note

  17. Scrapegraph-demo

    AI repositories

    Streamlit demo of Scrapegraph-ai for GPT4-hackaton

    Indexed GitHub, ScrapeGraphAI 120 ★MITUpdated yesterday

    git clone https://github.com/ScrapeGraphAI/Scrapegraph-demo.git
  18. seo-audit-console

    MCP servers

    Technical SEO in Claude: GSC + crawl + DataForSEO - audit, portfolio, market sizing, content recon.

    Stale Official MCP Registry, io.github.houtini-ai 101 ★Updated 4 days ago 2 notes

    @houtini/seo-audit-console@1.0.5
  19. linkedin-mcp-server

    MCP servers

    Open-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI agent access to profiles, companies, jobs, and messages.

    Indexed GitHub, stickerdaniel 4,226 ★Apache-2.0Updated today

    git clone https://github.com/stickerdaniel/linkedin-mcp-server.git
  20. Agent skills for fetching knowledge

    Stale GitHub, diffbot 5 ★MITUpdated 8 days ago

    git clone https://github.com/diffbot/diffbot-skills.git
  21. Black SEO Analyzer

    MCP servers

    Audit a page or crawl a whole site for technical SEO issues, locally. Compares crawls.

    Stale Official MCP Registry, io.github.sethblack Updated 12 days ago 2 notes

    https://github.com/sethblack/black-seo-analyzer/releases/download/v26.11.3/bseoa-26.11.3-win32.mcpb@26.11.3
  22. seo-ops

    Skills

    SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.

    Stale GitHub, tigerless-labs 951 ★Updated 21 days ago

    git clone https://github.com/tigerless-labs/seo-ops.git
  23. Siteiz

    MCP servers

    Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.

    Stale Official MCP Registry, com.siteiz Updated 44 days ago 1 note

  24. Commoncrawl

    MCP servers

    Common Crawl's public web archive — find every time a URL was crawled since 2008 and read the…

    Stale Official MCP Registry, io.github.pipeworx-io Updated 16 days ago 3 notes

    @pipeworx/mcp-commoncrawl@0.2.1

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.