Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
ketch
Agents & frameworksFast, stateless CLI for web search and scrape. Built for AI agents.
git clone https://github.com/1broseidon/ketch.git -
nopecha-extension
Agents & frameworksAutomated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.
git clone https://github.com/NopeCHALLC/nopecha-extension.git -
seo-crawler-mcp
MCP serversCrawl and analyse websites for SEO errors using Crawlee with SQLite storage
@houtini/seo-crawler-mcp@2.0.1 -
Crawl Readiness
Public APIsCheck what 50+ AI crawlers can read on a website and whether a page permits an AI use
-
decodo-openclaw-skill
AI repositoriesOpenClaw skill for scraping any URL using the Decodo Web Scraping API.
git clone https://github.com/Decodo/decodo-openclaw-skill.git -
donsetch
Agents & frameworksWeb fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
git clone https://github.com/dondai44423/donsetch.git -
crawl-readiness
MCP serversCheck what AI crawlers may read and what a page permits, generate llms.txt and schema.
crawl-readiness-mcp@0.2.9 -
llm-reader
Agents & frameworksTurn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.
git clone https://github.com/m92vyas/llm-reader.git -
Crawl Census
MCP serversAsk before you fetch: will this domain serve your crawler, refuse it, or charge it?
-
mcp-site-crawler-links-tables
MCP serversCrawl a website for links, file download links, HTML tables as rows, and lists, with no AI
@mambalabsdev/mcp-site-crawler-links-tables@1.0.2 -
JustHireMe
AI repositoriesLocal-first AI job intelligence workbench for scraping roles, ranking fit, and generating tailored application materials.
git clone https://github.com/vasu-devs/JustHireMe.git -
Website to Markdown Crawler
MCP serversCrawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
-
reverse-api-engineer
Agents & frameworksReverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor and OpenCode.
git clone https://github.com/kalil0321/reverse-api-engineer.git -
Flarum Forum Scraper - Discussion Crawler Tool
MCP serversSearch or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…
-
MinerU-HTML
AI repositoriesMinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.
git clone https://github.com/opendatalab/MinerU-HTML.git -
agentql-mcp
MCP serversModel Context Protocol server that integrates AgentQL's data extraction capabilities.
git clone https://github.com/tinyfish-io/agentql-mcp.git -
Upwork-AI-jobs-applier
Agents & frameworksAI tool for automating Upwork job applications using AI agents to find and qualify jobs, write personalized cover letters, and prepare for interviews based on your skills and experience.
git clone https://github.com/kaymen99/Upwork-AI-jobs-applier.git -
IndexLinks
MCP serversCheck how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.
-
google-maps-scraper-kit
SkillsFree, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
git clone https://github.com/Mahanaicoach/google-maps-scraper-kit.git -
google-search
MCP serversA Playwright-based Node.js tool that bypasses search engine anti-scraping mechanisms to execute Google searches. Local alternative to SERP APIs with MCP server integration.
git clone https://github.com/web-agent-master/google-search.git -
Skill_Seekers
SkillsConvert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git -
x-reader
MCP serversUniversal content reader MCP Server for 10+ platforms
git clone https://github.com/runesleo/x-reader.git -
xpay Web Scraping
MCP servers35+ web scraping tools. Firecrawl, Bright Data, Jina, Olostep, ScrapeGraph, Notte, Riveter.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.