Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
AI Crawler Index
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Crawler Log Triage
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Turn a website into Markdown with headings, links and word counts.
-
Crawler IP Verifier - real Googlebot or fake
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
-
crawlproof
MCP serversAI-crawler audits + ads (crawlproof.com)
-
IndexLinks
MCP serversCheck how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.
-
Leaked-GPTs
Docs & tutorialsLeaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
git clone https://github.com/friuns2/Leaked-GPTs.git -
browserpaw
MCP serversBrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
git clone https://github.com/GoldenLoaf24h/browserpaw.git -
Crawlwise
MCP serversSEO audits and AI crawler checks. Honest findings only; no invented scores.
-
Crawlerpeek
MCP serversCheck whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
crawlerpeek@0.3.1 -
Siteiz
MCP serversCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
-
fess
AI repositoriesOpen-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
git clone https://github.com/codelibs/fess.git -
Crawllama
AI repositoriesCrawlLama 🦙 is an local AI agent that answers questions via Ollama and integrates web- and RAG-based research.
git clone https://github.com/arn-c0de/Crawllama.git -
UAParse
MCP serversParse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.
-
Crawl4Agent
MCP serverscrawl4ai-compatible web crawler API: POST a URL, get clean LLM-ready Markdown with fit_markdown...
-
olostep-mcp-server
MCP serversMCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
git clone https://github.com/olostep/olostep-mcp-server.git -
diffbot-skills
SkillsAgent skills for fetching knowledge
git clone https://github.com/diffbot/diffbot-skills.git -
Crawl Readiness
Public APIsCheck what 50+ AI crawlers can read on a website and whether a page permits an AI use
-
crawl-readiness
MCP serversCheck what AI crawlers may read and what a page permits, generate llms.txt and schema.
crawl-readiness-mcp@0.2.9 -
Crawl Census
MCP serversAsk before you fetch: will this domain serve your crawler, refuse it, or charge it?
-
Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402.
-
facebook-ads-library-mcp
MCP serversFree, self-hosted MCP server that scrapes the public Facebook Ad Library (no API token) for competitive ad research — advertiser, copy, landing pages, CTAs, any country.
git clone https://github.com/RamsesAguirre777/facebook-ads-library-mcp.git -
Apify MCP server
MCP serversLets an agent run Apify's ready-made scrapers to pull data from websites, search engines, maps and social platforms.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.