Crawl Census
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
Open the source (opens github.com) https://github.com/taylorsmithgg/crawl-census-client
Before you use it
- Runs on someone else's server: what you send it leaves your machine.
What it is
MCP servers from io.github.taylorsmithgg.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Remote endpoint (streamable-http)
https://crawlcensus.com/mcp
From the server's entry in the official MCP Registry
Related
-
webclaw
MCP serversFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
git clone https://github.com/0xMassi/webclaw.git -
firecrawl-mcp-server
MCP serversOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.git -
fastCRW
MCP serversScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
-
Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay
-
olostep-mcp-server
MCP serversMCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
git clone https://github.com/olostep/olostep-mcp-server.git -
seo-crawler-mcp
MCP serversCrawl and analyse websites for SEO errors using Crawlee with SQLite storage
@houtini/seo-crawler-mcp@2.0.1