InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

  1. distill-mcp

    MCP servers

    Web pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.

    Stale Official MCP Registry, io.github.Bytestay Updated 4 days ago 2 notes

    bytestay-distill-mcp@0.2.9
  2. ai-visibility

    MCP servers

    AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

    Stale Official MCP Registry, com.lastminutedealshq Updated 33 days ago 1 note

  3. agent-ready-kit

    MCP servers

    Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.

    Stale Official MCP Registry, com.mudko Updated 3 months ago 1 note

  4. Indexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.

    Stale Official MCP Registry, com.stillindexed Updated 22 days ago 1 note

  5. tdm-reservation-lint

    MCP servers

    Finds the robots.txt lines that reserve nothing — retired crawler tokens, shadowed groups, rules abo

    Stale Official MCP Registry, io.github.jmshinhwa Updated 54 days ago 1 note

    @readystack/tdm-reservation-lint@1.0.5
  6. AI Visibility Checker

    MCP servers

    AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.

    Stale Official MCP Registry, io.github.krisdiallo Updated 2 months ago 1 note

    https://github.com/krisdiallo/ecom-agent/releases/download/v1.5.3/aivis.mcpb@1.5.3
  7. polite-crawler

    MCP servers

    A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

    Indexed Official MCP Registry, io.github.mark1russell7 Updated yesterday 1 note

    polite-crawler-mcp@0.7.6
  8. robotstxt-ai

    MCP servers

    Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

    Stale Official MCP Registry, io.github.sharozdawa Updated 8 months ago 1 note

    robotstxt-ai-mcp@1.0.1
  9. QueryRev AI Readiness

    MCP servers

    Checks a site for AI crawler access, robots.txt and llms.txt. Runs locally, no telemetry.

    Stale Official MCP Registry, com.queryrev Updated 22 days ago 1 note

    @queryrev/ai-readiness-mcp@1.0.0
  10. Doorman

    MCP servers

    Check which AI bots a website or a pasted robots.txt lets in, grouped by purpose.

    Stale Official MCP Registry, com.clawhard Updated 7 days ago 1 note

  11. SEOClave

    MCP servers

    Google keyword data for Latin America plus technical SEO, robots.txt, DNS and AI-crawler checks.

    Stale Official MCP Registry, com.seoclave Updated 4 days ago 1 note

  12. llms.txt 檢查工具

    MCP servers

    檢查網站 llms.txt 與 AI 爬蟲 robots.txt 設定,並自動產生 llms.txt。

    Stale Official MCP Registry, dev.workers.panda198271.llms-check Updated 50 days ago 1 note

  13. Page Fetch

    MCP servers

    Read one public web page as clean Markdown with its links, after a robots.txt check.

    Stale Official MCP Registry, app.openkrill Updated 26 days ago 1 note

  14. GEO SEO Records

    MCP servers

    Gets a website ready for AI assistants: robots.txt fixed, llms.txt drafted (access pass).

    Stale Official MCP Registry, com.seorecords Updated 7 days ago 1 note

  15. RobotsGate

    MCP servers

    Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

    Stale Official MCP Registry, dev.workers.mike-tusa.robotsgate Updated 31 days ago 1 note

  16. WebRead

    MCP servers

    Turn any public web page into clean Markdown for LLMs; respects robots.txt. By Crypto Money Maker

    Stale Official MCP Registry, co.youraitools Updated 26 days ago 1 note

  17. ai-seo

    MCP servers

    MCP server for AI-SEO auditing: schema, robots.txt, llms.txt, citation scoring, and AEO/GEO rewrites

    Stale Official MCP Registry, io.github.AutomateLab-tech Updated 6 months ago 1 note

    @automatelab/ai-seo-mcp@0.1.1
  18. NetLens

    MCP servers

    Read any web page as clean, ad-stripped Markdown past robots.txt and bot blocks, plus web search.

    Stale Official MCP Registry, io.github.pzalutski-pixel Updated 3 months ago 1 note

    netlens-mcp@0.2.3
  19. Agent Crawl Readiness

    MCP servers

    Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

    Stale Official MCP Registry, io.github.sadri-dridi Updated 54 days ago 1 note

  20. web-analytics-mcp

    MCP servers

    Web Analitiği MCP sunucusu — in-house on-page SEO denetimi (robots.txt + SSRF korumalı crawl),…

    Stale Official MCP Registry, io.github.sercanmetalore Updated 3 months ago 1 note

  21. Citehound

    MCP servers

    Scan any domain for AI readiness and generate llms.txt, robots.txt and schema. Free, read-only.

    Stale Official MCP Registry, io.github.andacuzel Updated 25 days ago 1 note

  22. Checks robots.txt for AI search vs training crawlers, llms.txt, JSON-LD, canonicals and sitemaps.

    Stale Official MCP Registry, io.github.alicexixi85 Updated 6 days ago 1 note

    ghcr.io/alicexixi85/geo-seo-mcp-pro:0.1.0
  23. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 1 note

  24. Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

    Stale Official MCP Registry, com.actablesite Updated 4 months ago 2 notes

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.