Web ingestion and crawling
Crawlers, scrapers, and web ingestion pipelines primarily collecting or extracting web content for downstream use. Includes browser-driven and AI-assisted collection. Excludes general interactive browser agents, standalone document-format conversion, and complete retrieval-and-answering applications.
| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| unclecode/crawl4ai | Crawls websites and converts their content into clean, LLM-ready Markdown for downstream AI use. | ★ 85.8K | Python | Apache-2.0 | v0.9.5 |