Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Check declared robots.txt permissions for six AI crawlers at URL paths
-
Technical SEO audit crawler: broken links, 404s, redirects, titles, canonicals. Runs on Apify.
-
AI Crawler Access Check by Citant.ai
MCP serversCan ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.
-
Crimson Crawler
MCP serversEnrich, search, assess, and manage threat intelligence through 80+ typed MCP tools.
-
Find broken links on a website: crawls its pages and reports 404s, errors and redirect chains
-
Ultra-FineWeb-L1
Datasets & benchmarksHugging Face dataset by openbmb
openbmb/Ultra-FineWeb-L1 -
smart-crawler
MCP serversE-commerce competitor intelligence, VOC sentiment, Amazon ASIN analysis, Reddit growth playbooks.
-
Code Crawler
MCP serversStatic and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
@nithin554/code-crawler@0.1.92 -
dingo
AI repositoriesDingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
git clone https://github.com/MigoXLab/dingo.git -
browserpaw
MCP serversBrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
git clone https://github.com/GoldenLoaf24h/browserpaw.git -
The Crawler Zoo Arena
MCP serversGames for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.
-
AI Crawler Index
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Crawler Log Triage
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
Turn a website into Markdown with headings, links and word counts.
-
Crawler IP Verifier - real Googlebot or fake
MCP servers{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
-
One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.
-
Scrapegraph-demo
AI repositoriesStreamlit demo of Scrapegraph-ai for GPT4-hackaton
git clone https://github.com/ScrapeGraphAI/Scrapegraph-demo.git -
seo-audit-console
MCP serversTechnical SEO in Claude: GSC + crawl + DataForSEO - audit, portfolio, market sizing, content recon.
@houtini/seo-audit-console@1.0.5 -
linkedin-mcp-server
MCP serversOpen-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI agent access to profiles, companies, jobs, and messages.
git clone https://github.com/stickerdaniel/linkedin-mcp-server.git -
diffbot-skills
SkillsAgent skills for fetching knowledge
git clone https://github.com/diffbot/diffbot-skills.git -
Black SEO Analyzer
MCP serversAudit a page or crawl a whole site for technical SEO issues, locally. Compares crawls.
https://github.com/sethblack/black-seo-analyzer/releases/download/v26.11.3/bseoa-26.11.3-win32.mcpb@26.11.3 -
seo-ops
SkillsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
git clone https://github.com/tigerless-labs/seo-ops.git -
Siteiz
MCP serversCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
-
Commoncrawl
MCP serversCommon Crawl's public web archive — find every time a URL was crawled since 2008 and read the…
@pipeworx/mcp-commoncrawl@0.2.1
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.