InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

  1. robots.txt group count

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  2. robots.txt Allow count, body discarded

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  3. robots-txt-checker

    MCP servers

    Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).

    Stale Official MCP Registry, io.github.ryuamarines Updated 18 days ago 1 note

  4. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  5. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  6. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  7. {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  8. Hosted MCP: CSV/XLSX to JSON rows and text-PDF pages; private uploads or URLs, no API key.

    Stale Official MCP Registry, com.forevenza.services Updated 9 days ago 1 note

  9. Scrape-LE

    MCP servers

    Analyse robots.txt content and report whether a path may be crawled.

    Stale Official MCP Registry, io.github.nolindnaidoo Updated 20 days ago 1 note

    scrape-le-mcp@2.6.0
  10. matchstick

    MCP servers

    Checks a site build: sitemaps, structured data, robots.txt and what crawlers see.

    Stale Official MCP Registry, io.github.Astralchemist Updated 23 days ago 1 note

    @astralchemist/matchstick@0.1.0-alpha.4
  11. Robots Inspect

    MCP servers

    Fetch a public robots.txt and return group counts only.

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  12. SEOAudit

    MCP servers

    On-page SEO analysis, HTTP header checks, broken link detection, and robots.txt parsing.

    Stale Official MCP Registry, io.github.therealMrFunGuy Updated 7 months ago 1 note

  13. marsgeo

    MCP servers

    Generate SEO structured data: JSON-LD schema, robots.txt, sitemaps, hreflang, Open Graph, meta.

    Stale Official MCP Registry, io.github.cnmars Updated 43 days ago 1 note

  14. url2md

    MCP servers

    PDFs, JavaScript pages and ordinary HTML as clean Markdown. Follows robots.txt per RFC 9309.

    Stale Official MCP Registry, io.url2md Updated 2 months ago 3 notes

    @url2md-io/mcp@0.1.5
  15. PriceMoved

    MCP servers

    Waitlist-stage pricing-page monitor: plans and early pricing, plus a PriceMovedBot robots.txt check.

    Stale Official MCP Registry, com.pricemoved Updated 32 days ago 1 note

  16. Botcard

    MCP servers

    Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.

    Stale Official MCP Registry, com.whosthatbot Updated 28 days ago 1 note

  17. Site Search Opener

    MCP servers

    SEO and competitor research from sitemaps and feeds, robots.txt respected, Google link fallback.

    Stale Official MCP Registry, io.github.albertalbs Updated 20 days ago 1 note

    site-search-opener-mcp@1.1.1
  18. citable

    MCP servers

    Can ChatGPT, Claude and Perplexity read and cite your site? robots.txt, edge blocks, llms.txt.

    Stale Official MCP Registry, io.github.genzagents Updated 19 days ago 1 note

    citable-cli@0.4.2
  19. distill-mcp

    MCP servers

    Web pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.

    Stale Official MCP Registry, io.github.hybridforgelab Updated 10 days ago 3 notes

    hybridforgelab-distill-mcp@0.2.3
  20. geoscore

    MCP servers

    AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.

    Stale Official MCP Registry, io.github.CloudCorpRecords Updated 13 days ago 1 note

    @reneturcios/geoscore@0.1.0
  21. distill-mcp

    MCP servers

    Web pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.

    Stale Official MCP Registry, io.github.Bytestay Updated 4 days ago 2 notes

    bytestay-distill-mcp@0.2.9
  22. ai-visibility

    MCP servers

    AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

    Stale Official MCP Registry, com.lastminutedealshq Updated 33 days ago 1 note

  23. agent-ready-kit

    MCP servers

    Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.

    Stale Official MCP Registry, com.mudko Updated 3 months ago 1 note

  24. Indexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.

    Stale Official MCP Registry, com.stillindexed Updated 22 days ago 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.