Library
85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
Crawl4AI
Public APIsAn open-source crawler that turns websites into Markdown ready for an AI model.
pip install -U crawl4ai -
Firecrawl
Public APIsTurns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.
-
Crawl Readiness
Public APIsCheck what 50+ AI crawlers can read on a website and whether a page permits an AI use
-
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
ProxyCrawl
Public APIsScraping and crawling anticaptcha service
-
Webclaw
Public APIsWeb content extraction for LLMs with scrape, crawl, search, and summarize
-
RealBots
Public APIsVerify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real
-
CitedBy
Public APIsWhich AI crawlers can read a site: robots.txt and server answer for each
-
SnoopScan
Public APIsScrape, crawl, map and search the web as clean markdown or JSON, with a free monthly tier
-
TidyTools
Public APIsWeb page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)
-
ContextFetch
Public APIsWeb search and any page as Markdown pruned to the query, plus research, crawl and site maps
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.