InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

    Stale Official MCP Registry, io.github.Nero-Engine Updated 15 days ago 1 note

  2. crawlbrulee

    Public APIs

    EU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more

    Indexed public-apis (GitHub list)

  3. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  4. Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.

    Stale Official MCP Registry, io.github.stellalytics Updated 3 days ago 1 note

  5. browserpaw

    MCP servers

    BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus

    Stale GitHub, GoldenLoaf24h 38 ★AGPL-3.0Updated 14 days ago

    git clone https://github.com/GoldenLoaf24h/browserpaw.git
  6. olostep-mcp-server

    MCP servers

    MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.

    Stale GitHub, olostep 25 ★MITUpdated 6 days ago

    git clone https://github.com/olostep/olostep-mcp-server.git
  7. Can ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.

    Stale Official MCP Registry, io.github.hamzawamiqhq Updated 4 days ago 1 note

  8. Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.

    Stale Official MCP Registry, io.github.ByteBlisk Updated 8 days ago 1 note

  9. Crimson Crawler

    MCP servers

    Enrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.

    Stale Official MCP Registry, com.crimsoncrawler Updated 3 months ago 1 note

  10. basemind

    AI repositories

    Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

    Indexed GitHub, Goldziher 121 ★MITUpdated today

    git clone https://github.com/Goldziher/basemind.git
  11. Free, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research — advertiser, copy, landing pages, CTAs, any country.

    Stale GitHub, RamsesAguirre777 273 ★MITUpdated 2 months ago

    git clone https://github.com/RamsesAguirre777/facebook-ads-library-mcp.git
  12. Find broken links on a website: crawls its pages and reports 404s, errors and redirect chains

    Stale Official MCP Registry, io.github.kriton-dev Updated 14 days ago 1 note

  13. smart-crawler

    MCP servers

    E-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.

    Stale Official MCP Registry, io.github.shulex-voc Updated 5 months ago 1 note

  14. crawlbrulee

    Public APIs

    EU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more

    Indexed public-apis (GitHub list) 1 note

  15. Code Crawler

    MCP servers

    Static and dynamic call tracing MCP server backed by a SQLite property graph of your codebase

    Indexed Official MCP Registry, io.github.nithin554 Updated yesterday 1 note

    @nithin554/code-crawler@0.1.92
  16. CrawlCheck

    MCP servers

    Verification layer for the agentic web: a signed answer about any site before an agent acts.

    Stale Official MCP Registry, io.github.emmanuelorta Updated 2 days ago 1 note

  17. CrawlEyes MCP Server

    MCP servers

    Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.

    Stale Official MCP Registry, io.github.waiky-github Updated 2 months ago 2 notes

    crawleyes@0.2.0
  18. Search or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…

    Stale Official MCP Registry, com.ranovam Updated 49 days ago 1 note

  19. crawlbit

    MCP servers

    Four free tools to check if AI engines can find, read and recognise a site. No API key.

    Stale Official MCP Registry, io.github.amati032-dev Updated 3 months ago 1 note

    crawlbit-mcp@1.0.1
  20. AI Crawler Index

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  21. Crawler Log Triage

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  22. 🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

    Stale GitHub, Evil0ctal 21.1k ★Apache-2.0Updated 14 days ago

    git clone https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git
  23. Turn a website into Markdown with headings, links and word counts.

    Stale Official MCP Registry, io.github.bepolabs Updated 4 days ago 1 note

  24. IndexLinks

    MCP servers

    Check how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.

    Stale Official MCP Registry, app.indexlinks Updated 42 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.