InnerVault

Crawl Readiness

Indexedpublic-apis (GitHub list)

Check what 50+ AI crawlers can read on a website and whether a page permits an AI use

Open the source (opens crawlreadiness.com) https://crawlreadiness.com

What it is

Public APIs from public-apis (GitHub list).

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

How to get it

Setup steps: not provided by the source. Follow the instructions on the source.

Related

  1. Crawl4AI

    Public APIs

    An open-source crawler that turns websites into Markdown ready for an AI model.

    Hand-checked InnerVault (hand-checked), unclecode 87.3k ★Apache-2.0

    pip install -U crawl4ai
  2. ProxyCrawl

    Public APIs

    Scraping and crawling anticaptcha service

    Indexed public-apis (GitHub list)

  3. Firecrawl

    Public APIs

    Turns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.

    Hand-checked InnerVault (hand-checked), Firecrawl 201.9k ★AGPL-3.0

  4. Webclaw

    Public APIs

    Web content extraction for LLMs with scrape, crawl, search, and summarize

    Indexed public-apis (GitHub list)

  5. SnoopScan

    Public APIs

    Scrape, crawl, map and search the web as clean markdown or JSON, with a free monthly tier

    Indexed public-apis (GitHub list)

  6. Scrax

    Public APIs

    Web scraping API that only bills for successful scrapes

    Indexed public-apis (GitHub list)