Crawl4AI
An open-source crawler that turns websites into Markdown ready for an AI model.
Open the source (opens github.com) https://github.com/unclecode/crawl4ai
Before you use it
- Respect robots.txt, site terms and privacy law.
What it is
Public APIs from unclecode. Good for:
- Collect pages for research without a paid API
Level: builder
How to get it
Install
pip install -U crawl4ai
Copied from the official README or docs
Related
-
Firecrawl
Public APIsTurns websites into clean Markdown or structured data for AI, through a hosted API or a self-hosted copy.
-
TidyTools
Public APIsWeb page to clean Markdown and AI crawler robots.txt checks, free without a key (rate limited)
-
GcrawlAI
Public APIsWeb scraping, SERP API, screenshots, URL to Markdown and prebuilt extractors
-
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more
-
SnoopScan
Public APIsScrape, crawl, map and search the web as clean markdown or JSON, with a free monthly tier
-
crawlbrulee
Public APIsEU-native web scraping for AI agents & developers: turn any page into markdown, screenshots and more