RAG-BenchLab (sgyaqing/rag-benchlab) is a tool that builds RAG evaluation test sets from documents and grades the answers of multiple RAG systems. The README states that it provides five RAG metrics and result caching. The version was bumped to 0.1.1 with the 2026-10-08 commit, and the center of this release is making the model settings connection check (connection confirmation) survive silent failure through retries. The change was 261 lines added and 30 lines deleted across 8 files.
The connection check can end in a ReadTimeout because there is a roughly 10-second interval in which the request is only accepted and no response comes back. The commit message says that such an interval was observed during the tool's own pre-check. 0.1.1 retries up to 3 times using exponential backoff, but authentication and billing responses such as 401, 402, and 403 are excluded from the retry targets, so that a wrong key fails immediately. The overall time limit is 30 seconds; the first attempt uses the existing 15-second read and 10-second connection times as they are, and subsequent attempts are allotted only the remaining time. A request-unit timeout argument was added to chat_completion, and the llm_http test grew by 148 lines. UI screenshots were added to both READMEs, and the Docker image tag and the distribution file names were also changed to 0.1.1.
However, even though the release has changed, the authentication feature is still absent. The README warns that 0.1.1 also has no login handling, so anyone who can access it can see the stored API keys, and therefore it should be placed on an internal network only or behind an authenticating proxy. The record of non-response intervals is the maintainer's own observation, and the impact of retries on evaluation scores or test set quality is not presented. Judging by this, the change appears to be of a nature that reduces false positives at the settings-check stage, and does not touch the evaluation criteria themselves.