unclecode/crawl4ai v0.9.5 is a Python-based open-source web crawler and scraper for LLMs and AI agents. It converts website content into Markdown that LLMs can easily use, and is released under the Apache-2.0 license. The repository has 85,644 GitHub stars.
For running it yourself, you install a browser with `pip install -U crawl4ai` and `crawl4ai-setup`, then use the async crawler AsyncWebCrawler. The README presents three usage options: the free library, a self-hosted Docker server, and Crawl4AI Cloud. The hosted server exposes `/scrape`, which returns Markdown, and `/extract`, which returns JSON, along with search, question-and-answer, and multi-URL processing paths. `/scrape/jobs` is the job path that handles multiple URLs, and an MCP endpoint is also provided. The documentation also describes an async browser pool with caching, adaptive handling that explores only the parts needed for a given site pattern, and session, proxy, and cookie control.
Crawl4AI Cloud offers up to 1,000 credits per month for free, with paid usage starting at $10 per month. The library, the self-hosted server, and the hosted service are therefore options that differ in operation and cost. The provided material contains no figures comparing the performance of these paths. In particular, the claim that it has established itself as the standard for agent data-collection layers is not backed by any comparison results, so its real-world suitability has to be judged separately.