Website Crawler — Markdown Pages, Sitemaps & RAG Provenance
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
Open the source (opens apify.com) https://apify.com/hermeskd/website-to-markdown
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from io.github.kriton-dev.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://mcp.apify.com/?tools=hermeskd/website-to-markdown
From the server's entry in the official MCP Registry
Related
-
Web Scraper & Crawler for RAG (Markdown)
MCP serversWebsite to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
-
Apify MCP server
MCP serversLets an agent run Apify's ready-made scrapers to pull data from websites, search engines, maps and social platforms.
-
Playwright MCP
MCP serversLets an agent drive a real browser: open pages, click, fill in forms and read what's on screen.
claude mcp add playwright npx @playwright/mcp@latest -
Website to Markdown Crawler
MCP serversCrawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
-
Turn a website into Markdown with headings, links and word counts.
-
doc-scraper
MCP serversGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
git clone https://github.com/Sriram-PR/doc-scraper.git