Library
85,762 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
-
AI Crawler Access Check by Citant.ai
MCP serversCan ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.
-
Crimson Crawler
MCP serversEnrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.
-
The Crawler Zoo Arena
MCP serversGames for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
-
mcp-site-crawler-links-tables
MCP serversCrawl a website for links, file download links, HTML tables as rows, and lists, with no AI
@mambalabsdev/mcp-site-crawler-links-tables@1.0.2 -
smart-crawler
MCP serversE-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
Website to Markdown Crawler
MCP serversCrawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
-
Code Crawler
MCP serversStatic and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
@nithin554/code-crawler@0.1.92 -
Flarum Forum Scraper - Discussion Crawler Tool
MCP serversSearch or monitor any Flarum-powered forum for discussions, replies, participants, dates, and…
-
Web Scraper & Crawler for RAG (Markdown)
MCP serversWebsite to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
-
AI Crawler Index
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Crawler Log Triage
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Turn a website into Markdown with headings, links and word counts.
-
Crawler IP Verifier - real Googlebot or fake
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
-
crawlproof
MCP serversAI-crawler audits + ads (crawlproof.com)
-
IndexLinks
MCP serversCheck how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts.
-
browserpaw
MCP serversBrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
git clone https://github.com/GoldenLoaf24h/browserpaw.git -
Crawlwise
MCP serversSEO audits and AI crawler checks. Honest findings only; no invented scores.
-
Crawlerpeek
MCP serversCheck whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.
crawlerpeek@0.3.1 -
Siteiz
MCP serversCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
-
UAParse
MCP serversParse a User-Agent: browser, OS, device, plus bot, search crawler, AI crawler and AI agent flags.
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.