trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Open the source (opens github.com) https://github.com/adbar/trafilatura
What it is
AI repositories from adbar.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
How to get it
Get the code
git clone https://github.com/adbar/trafilatura.git
Standard git access to a public repository
Setup steps: not provided by the source. Follow the instructions on the source.
Related
-
MinerU-HTML
AI repositoriesMinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.
git clone https://github.com/opendatalab/MinerU-HTML.git -
Scrapegraph-ai
AI repositoriesPython scraper based on AI
git clone https://github.com/ScrapeGraphAI/Scrapegraph-ai.git -
searxNcrawl
AI repositoriesOpen-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.
git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git -
wigolo
AI repositoriesThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
git clone https://github.com/KnockOutEZ/wigolo.git -
fess
AI repositoriesOpen-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
git clone https://github.com/codelibs/fess.git -
AnyCrawl
AI repositoriesAnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
git clone https://github.com/any4ai/AnyCrawl.git