InnerVault

Crawl4AI

Hand-checked InnerVault (hand-checked), unclecode

An open-source crawler that turns websites into Markdown ready for an AI model.

Open the source (opens github.com) https://github.com/unclecode/crawl4ai

Before you use it

  • Respect robots.txt, site terms and privacy law.

What it is

Public APIs from unclecode. Good for:

  • Collect pages for research without a paid API

Level: builder

How to get it

Install

pip install -U crawl4ai

Copied from the official README or docs

Related

  1. Firecrawl

    Public APIs

    Turns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.

    Hand-checked InnerVault (hand-checked), Firecrawl 201.9k ★AGPL-3.0

  2. TidyTools

    Public APIs

    Web page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)

    Indexed public-apis (GitHub list)

  3. GcrawlAI

    Public APIs

    Web scraping, SERP API, screenshots, URL to Markdown and prebuilt extractors

    Indexed public-apis (GitHub list)

  4. crawlbrulee

    Public APIs

    EU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more

    Indexed public-apis (GitHub list)

  5. SnoopScan

    Public APIs

    Scrape, crawl, map and search the web as clean markdown or JSON, with a free monthly tier

    Indexed public-apis (GitHub list)

  6. crawlbrulee

    Public APIs

    EU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more

    Indexed public-apis (GitHub list) 1 note