Library
85,762 resources from 11 public sources. 53 checked by hand; the rest are unreviewed.
Results
Looks like a goal we have a starting path for: I want to research potential clients
-
RealBots
Public APIsVerify whether requests claiming to be GPTBot, ClaudeBot & other AI crawlers are real
-
CitedBy
Public APIsWhich AI crawlers can read a site: robots.txt and server answer for each
-
TidyTools
Public APIsWeb page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)
-
ScraperApi
Public APIsEasily build scalable web scrapers
-
ContextFetch
Public APIsWeb search and any page as Markdown pruned to the query, plus research, crawl and site maps
Indexed entries aren't reviewed or tested by InnerVault. Details come straight from each source. Anything that can reach your accounts, files or browser can do damage, so read the source before you install it.