InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. AI Crawler Index

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  2. Crawler Log Triage

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  3. Turn a website into Markdown with headings, links and word counts.

    Stale Official MCP Registry, io.github.bepolabs Updated 4 days ago 1 note

  4. {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  5. One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

    Stale Official MCP Registry, app.rowb.ai-visibility Updated 54 days ago 1 note

  6. crawlproof

    MCP servers

    AI-crawler audits + ads (crawlproof.com)

    Stale Official MCP Registry, com.profullstack Updated 5 days ago 1 note

  7. IndexLinks

    MCP servers

    Check how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.

    Stale Official MCP Registry, app.indexlinks Updated 42 days ago 1 note

  8. Leaked-GPTs

    Docs & tutorials

    Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

    Stale GitHub, friuns2 2,475 ★Updated 2 years ago

    git clone https://github.com/friuns2/Leaked-GPTs.git
  9. browserpaw

    MCP servers

    BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus

    Stale GitHub, GoldenLoaf24h 38 ★AGPL-3.0Updated 14 days ago

    git clone https://github.com/GoldenLoaf24h/browserpaw.git
  10. Crawlwise

    MCP servers

    SEO audits and AI crawler checks. Honest findings only; no invented scores.

    Stale Official MCP Registry, site.crawlwise Updated 11 days ago 1 note

  11. Crawlerpeek

    MCP servers

    Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.

    Stale Official MCP Registry, io.github.crawlerpeek Updated 18 days ago 1 note

    crawlerpeek@0.3.1
  12. Siteiz

    MCP servers

    Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.

    Stale Official MCP Registry, com.siteiz Updated 44 days ago 1 note

  13. fess

    AI repositories

    Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

    Indexed GitHub, codelibs 1,151 ★Apache-2.0Updated yesterday

    git clone https://github.com/codelibs/fess.git
  14. Crawllama

    AI repositories

    CrawlLama 🦙 is an local AI agent that answers questions via Ollama and integrates web- and RAG-based research.

    Indexed GitHub, arn-c0de 109 ★Custom licence (see repository)Updated 3 days ago

    git clone https://github.com/arn-c0de/Crawllama.git
  15. UAParse

    MCP servers

    Parse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.

    Stale Official MCP Registry, dev.workers.mike-tusa.uaparse Updated 15 days ago 1 note

  16. Crawl4Agent

    MCP servers

    crawl4ai-compatible web crawler API: POST a URL, get clean LLM-ready Markdown with fit_markdown...

    Stale Official MCP Registry, com.saastemly Updated 44 days ago 1 note

  17. olostep-mcp-server

    MCP servers

    MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.

    Stale GitHub, olostep 25 ★MITUpdated 6 days ago

    git clone https://github.com/olostep/olostep-mcp-server.git
  18. Agent skills for fetching knowledge

    Stale GitHub, diffbot 5 ★MITUpdated 8 days ago

    git clone https://github.com/diffbot/diffbot-skills.git
  19. Crawl Readiness

    Public APIs

    Check what 50+ AI crawlers can read on a website and whether a page permits an AI use

    Indexed public-apis (GitHub list)

  20. crawl-readiness

    MCP servers

    Check what AI crawlers may read and what a page permits, generate llms.txt and schema.

    Stale Official MCP Registry, io.github.Grubbomatic Updated 17 days ago 2 notes

    crawl-readiness-mcp@0.2.9
  21. Crawl Census

    MCP servers

    Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

    Stale Official MCP Registry, io.github.taylorsmithgg Updated 2 months ago 1 note

  22. Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.

    Stale Official MCP Registry, dev.oassis Updated 25 days ago 1 note

  23. Free, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research — advertiser, copy, landing pages, CTAs, any country.

    Stale GitHub, RamsesAguirre777 273 ★MITUpdated 2 months ago

    git clone https://github.com/RamsesAguirre777/facebook-ads-library-mcp.git
  24. Apify MCP server

    MCP servers

    Lets an agent run Apify's ready-made scrapers to pull data from websites, search engines, maps and social platforms.

    Hand-checked InnerVault (hand-checked), Apify 16.0k ★MIT

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.