InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

  1. robots.txt group count

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  2. robots.txt Allow count, body discarded

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  3. robots-txt-checker

    MCP servers

    Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).

    Stale Official MCP Registry, io.github.ryuamarines Updated 18 days ago 1 note

  4. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

    Stale Official MCP Registry, io.github.jmshinhwa Updated 58 days ago 1 note

    @readystack/robots-txt-ai-crawler-audit@0.1.3
  5. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

    Stale Official MCP Registry, dev.workers.quietworklabs.robots-check Updated 29 days ago 1 note

  6. Check declared robots.txt permissions for six AI crawlers at URL paths

    Stale Official MCP Registry, io.github.kriton-dev Updated 13 days ago 1 note

  7. --- name: robots-txt-and-sitemap-checker description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards…

    Indexed prompts.chat (CC0 prompt collection), f CC0-1.0

  8. {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

    Stale Official MCP Registry, dev.workers.pathwren.www Updated 2 months ago 1 note

  9. Open-source SEO & GEO audit: a zero-dependency CLI (npx), an agent skill for Claude Code, Cursor, Codex, Gemini & Copilot, 47 SEO prompts and a cited Google/Bing/AI-search knowledge base. Checks robots.txt, sitemaps, canonicals, hreflang, structured data, Core Web Vitals and AI crawlers — then fixes them.

    Indexed GitHub, umutxyp 581 ★MITUpdated 5 days ago

    git clone https://github.com/umutxyp/Seo-Promt-Master.git
  10. Hosted MCP: CSV/XLSX to JSON rows and text-PDF pages; private uploads or URLs, no API key.

    Stale Official MCP Registry, com.forevenza.services Updated 9 days ago 1 note

  11. CitedBy

    Public APIs

    Which AI crawlers can read a site: robots.txt and server answer for each

    Indexed public-apis (GitHub list)

  12. Scrape-LE

    MCP servers

    Analyse robots.txt content and report whether a path may be crawled.

    Stale Official MCP Registry, io.github.nolindnaidoo Updated 20 days ago 1 note

    scrape-le-mcp@2.6.0
  13. matchstick

    MCP servers

    Checks a site build: sitemaps, structured data, robots.txt and what crawlers see.

    Stale Official MCP Registry, io.github.Astralchemist Updated 23 days ago 1 note

    @astralchemist/matchstick@0.1.0-alpha.4
  14. Robots Inspect

    MCP servers

    Fetch a public robots.txt and return group counts only.

    Stale Official MCP Registry, io.github.sadri-dridi Updated 2 months ago 1 note

  15. TidyTools

    Public APIs

    Web page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)

    Indexed public-apis (GitHub list)

  16. SEOAudit

    MCP servers

    On-page SEO analysis, HTTP header checks, broken link detection, and robots.txt parsing.

    Stale Official MCP Registry, io.github.therealMrFunGuy Updated 7 months ago 1 note

  17. marsgeo

    MCP servers

    Generate SEO structured data: JSON-LD schema, robots.txt, sitemaps, hreflang, Open Graph, meta.

    Stale Official MCP Registry, io.github.cnmars Updated 43 days ago 1 note

  18. url2md

    MCP servers

    PDFs, JavaScript pages and ordinary HTML as clean Markdown. Follows robots.txt per RFC 9309.

    Stale Official MCP Registry, io.url2md Updated 2 months ago 3 notes

    @url2md-io/mcp@0.1.5
  19. PriceMoved

    MCP servers

    Waitlist-stage pricing-page monitor: plans and early pricing, plus a PriceMovedBot robots.txt check.

    Stale Official MCP Registry, com.pricemoved Updated 32 days ago 1 note

  20. Botcard

    MCP servers

    Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.

    Stale Official MCP Registry, com.whosthatbot Updated 28 days ago 1 note

  21. Site Search Opener

    MCP servers

    SEO and competitor research from sitemaps and feeds, robots.txt respected, Google link fallback.

    Stale Official MCP Registry, io.github.albertalbs Updated 20 days ago 1 note

    site-search-opener-mcp@1.1.1
  22. citable

    MCP servers

    Can ChatGPT, Claude and Perplexity read and cite your site? robots.txt, edge blocks, llms.txt.

    Stale Official MCP Registry, io.github.genzagents Updated 19 days ago 1 note

    citable-cli@0.4.2
  23. distill-mcp

    MCP servers

    Web pages to clean Markdown for AI agents, honouring robots.txt and AI opt-outs.

    Stale Official MCP Registry, io.github.hybridforgelab Updated 10 days ago 3 notes

    hybridforgelab-distill-mcp@0.2.3
  24. geoscore

    MCP servers

    AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.

    Stale Official MCP Registry, io.github.CloudCorpRecords Updated 13 days ago 1 note

    @reneturcios/geoscore@0.1.0

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.