adbar/trafilatura
6,675 417PythonApache-2.0Updated 2 months ago
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
article-extractorcorpus-buildercorpus-toolscrawlerhtml-to-markdownhtml2textllmnews-aggregatornews-crawlernlpragreadabilityrss-feedscrapingteitext-cleaningtext-extractiontext-miningtext-preprocessingweb-scraping
Our verdict: worth using
CLUSTER WINNER (boilerplate removal). Main-content extraction to markdown, no browser, no API key - this is the token-efficiency half of webget: a 264KB gov page comes back as ~9k chars of prose. Shipped in claude-config/webget.
Filed under dev tooling in our directory.