InnerVault

Library

85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.

Results

Looks like a goal we have a starting path for: I want to research potential clients

  1. RealBots

    Public APIs

    Verify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real

    Indexed public-apis (GitHub list)

  2. CitedBy

    Public APIs

    Which AI crawlers can read a site: robots.txt and server answer for each

    Indexed public-apis (GitHub list)

  3. TidyTools

    Public APIs

    Web page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)

    Indexed public-apis (GitHub list)

  4. ScraperApi

    Public APIs

    Easily build scalable web scrapers

    Indexed public-apis (GitHub list)

  5. ContextFetch

    Public APIs

    Web search and any page as Markdown pruned to the query, plus research, crawl and site maps

    Indexed public-apis (GitHub list) 1 note

Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.