rayenmlayeh/jevcrawl is an MCP (Model Context Protocol) server and CLI that explores websites for AI agents and finds pages containing answers. On October 26, 2026, commit 7aed9b7, which merged PR #9 (audit-fixes), is the latest change in this repository and remains as-is on GitHub. This commit touched redirect scope, time limits, sitemap gzip handling, and cache write batching. The changes span 14 files, with 199 lines added and 121 lines deleted.
The key point is that the boundary determining how far the crawl can go has become clear. If the final URL after redirects goes outside the host being crawled, that page is discarded. In configurations that respect robots.txt (the file of crawler permission rules), it is skipped even if it redirects to a URL blocked by robots. Only the starting page is an exception and may redirect anywhere. Each page-processing stage is now wrapped in asyncio.timeout, so even with a slow server or a retry loop it cannot exceed max_seconds, and it stops with max_seconds as the halt reason. Reading robots is now always possible regardless of configuration, and is used to read the sitemap (a list of the site's URLs) even when robots is not respected.
The purpose of this project is to let agents find the pages they need on their own. The repository description claims that it finds answer pages twice as often as Crawl4AI, and that it is 10 times cheaper per answer than having Claude crawl directly. This change is not evidence for those figures; it is a fix to the rules that actually stop out-of-scope movement and timeouts. Cache writes in the decide module were changed to be handled in batches, and the crawl tests grew by 63 lines.
There are clear limits to the judgment. The 2x and 10x figures are ones the repository description itself puts forward, and this commit contains no measurements or benchmarks backing them. The detailed implementation of sitemap gzip handling and cache write batching is not present in the public record of the patch for those files, so it can only be confirmed from the file changes and the commit message description. Neither the license nor the implementation language is included in this record. Since there is still no basis for asserting that performance improved, it is a change worth taking a look at before checking the results directly.