InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Scrapegraph-ai

    AI repositories

    Python scraper based on AI

    Stale GitHub, ScrapeGraphAI 31.8k ★MITUpdated 41 days ago

    git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git
  2. ai-crawler-py

    Agents & frameworks

    Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

    Stale Archived GitHub, oxylabs 3,262 ★Updated 2 months ago 1 note

    git clone https://github.com/oxylabs/ai-crawler-py.git
  3. ai-crawler-policy

    MCP servers

    Cloudflare Workers MCP server: ai-crawler-policy

    Stale Official MCP Registry, io.github.lazymac2x Updated 6 months ago 1 note

  4. AI crawler checker: shows which page content GPTBot and ClaudeBot can't read without JavaScript.

    Stale Official MCP Registry, io.github.missiongrowth Updated 26 days ago 1 note

    unrendered-mcp@0.1.4
  5. Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.

    Stale Official MCP Registry, com.atalaya-ai Updated 13 days ago 1 note

  6. ai-crawler-readiness

    MCP servers

    Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index

    Stale Official MCP Registry, com.lumnika Updated 58 days ago 1 note

  7. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  8. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

  9. Audit webpage access for AI crawlers. API key and Premium or Partner plan required.

    Stale Official MCP Registry, io.github.newmediamadman Updated 54 days ago 1 note

  10. Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.

    Stale Official MCP Registry, app.openkrill Updated 26 days ago 1 note

  11. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  12. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  13. Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.

    Stale Official MCP Registry, io.github.Nero-Engine Updated 15 days ago 1 note

  14. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  15. AI Crawler Index

    MCP servers

    {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  16. Can ChatGPT, Claude and Perplexity crawlers reach a site? Real per-bot fetch, firewall/CDN blocks.

    Stale Official MCP Registry, io.github.hamzawamiqhq Updated 4 days ago 1 note

  17. One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes.

    Stale Official MCP Registry, app.rowb.ai-visibility Updated 54 days ago 1 note

  18. Siteiz

    MCP servers

    Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.

    Stale Official MCP Registry, com.siteiz Updated 44 days ago 1 note

  19. crawlproof

    MCP servers

    AI-crawler audits + ads (crawlproof.com)

    Stale Official MCP Registry, com.profullstack Updated 5 days ago 1 note

  20. geo-skills

    Skills

    9 open Claude skills for GEO / AEO: check if AI search can read a page, collect and analyze how often AI engines name a brand (with confidence ranges), plus schema parity, AI crawler log checks and migration guards. From the AI Visibility Census.

    Indexed GitHub, aivisibilitycensus 5 ★MITUpdated 2 days ago

    git clone https://github.com/aivisibilitycensus/geo-skills.git
  21. RealBots

    Public APIs

    Verify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real

    Indexed public-apis (GitHub list)

  22. CitedBy

    Public APIs

    Which AI crawlers can read a site: robots.txt and server answer for each

    Indexed public-apis (GitHub list)

  23. visprobe

    MCP servers

    AI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.

    Stale Official MCP Registry, io.github.CloudCorpRecords Updated 14 days ago 1 note

    @reneturcios/visprobe@0.1.0
  24. rattlesnakesbymail

    MCP servers

    Dated, sourced claims about AI crawlers and agents, offered as MCP tools.

    Stale Official MCP Registry, com.rattlesnakesbymail Updated 58 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.