InnerVault

Library

85,340 resources from 11 public sources. 54 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. Crawl4AI

    Public APIs

    An open-source crawler that turns websites into Markdown ready for an AI model.

    Hand-checked InnerVault (hand-checked), unclecode 87.3k ★Apache-2.0

    pip install -U crawl4ai
  2. Firecrawl

    Public APIs

    Turns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.

    Hand-checked InnerVault (hand-checked), Firecrawl 201.9k ★AGPL-3.0

  3. Crawl Readiness

    Public APIs

    Check what 50+ AI crawlers can read on a website and whether a page permits an AI use

    Indexed public-apis (GitHub list)

  4. crawlbrulee

    Public APIs

    EU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more

    Indexed public-apis (GitHub list)

  5. crawlbrulee

    Public APIs

    EU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more

    Indexed public-apis (GitHub list) 1 note

  6. ProxyCrawl

    Public APIs

    Scraping and crawling anticaptcha service

    Indexed public-apis (GitHub list)

  7. Webclaw

    Public APIs

    Web content extraction for LLMs with scrape, crawl, search, and summarize

    Indexed public-apis (GitHub list)

  8. RealBots

    Public APIs

    Verify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real

    Indexed public-apis (GitHub list)

  9. CitedBy

    Public APIs

    Which AI crawlers can read a site: robots.txt and server answer for each

    Indexed public-apis (GitHub list)

  10. SnoopScan

    Public APIs

    Scrape, crawl, map and search the web as clean markdown or JSON, with a free monthly tier

    Indexed public-apis (GitHub list)

  11. TidyTools

    Public APIs

    Web page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)

    Indexed public-apis (GitHub list)

  12. ContextFetch

    Public APIs

    Web search and any page as Markdown pruned to the query, plus research, crawl and site maps

    Indexed public-apis (GitHub list) 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.