Agent Crawl Readiness
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
Open the source (opens github.com) https://github.com/sadri-dridi/named-mcp-utilities
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from io.github.sadri-dridi.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp
From the server's entry in the official MCP Registry
Related
-
webclaw
MCP serversFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
git clone https://github.com/0xMassi/webclaw.git -
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
pagesift
MCP serversLocal-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, and verifiable data graphs.
git clone https://github.com/YOLOW92/pagesift.git -
fastCRW
MCP serversScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
-
graphlit-mcp-server
MCP serversModel Context Protocol (MCP) Server for Graphlit Platform
git clone https://github.com/graphlit/graphlit-mcp-server.git -
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git