github.devotd.
NewData

adbar/trafilatura v2.3.1 — web body and metadata extraction

adbar/trafilatura is a Python tool that extracts page bodies and metadata through web crawling and scraping and exports them as TXT, JSON, Markdown and other formats, and the latest GitHub release is v2.3.1.

Repositoryadbar/trafilatura ↗

adbar/trafilatura is a Python tool that extracts page bodies and metadata through web crawling and scraping and exports them as TXT, JSON, Markdown and other formats, and the latest GitHub release is v2.3.1.

GitHub description: Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

stars
★ 7K
Forks
461
Language
Python
License
Apache-2.0
Latest release
v2.3.1
Read original source ↗View on GitHub ↗

adbar/trafilatura is a Python package and command-line tool that collects text from the web and converts raw HTML into structured data. The repository description says it supports URL download, web crawling and scraping, and extracts the body, metadata and document annotations. The latest GitHub release is v2.3.1.

Crawling is walking a site to collect pages, and scraping is pulling out only the values you need from the collected pages. In addition to taking URLs directly, the tool supports discovery that finds links in web feed lists. It reduces noise such as repeated headers and footers, and keeps the body and metadata. Output formats include TXT, Markdown, CSV, JSON, HTML and XML. The implementation language is Python and the license is Apache-2.0. The repository was created on April 8, 2019 and has 6998 GitHub stars.

Still, this record alone limits how far the judgment can go. There is no changelog, so what changed in v2.3.1 cannot be known. No figures or criteria were given for comparing body extraction accuracy with other tools. The only remaining score is momentum 3.6, and the novelty, technical and quality scores are empty. Apart from the fact that the project has lasted, this record holds no evidence supporting extraction quality. So this record alone cannot decide adoption, and separate verification is needed before use.