ninjahawk/livenerf is a long-running benchmark that tracks how much a frontier model's capability drops after it is released. It is written in Python and released under the MIT license. It has 1544 GitHub stars, and the repository was created on September 22, 2026. There is no public release yet. What the repository publishes is a measurement record repeated every day, plus the procedure that decides each verdict.
The README states that its goal is to measure, in a reproducible way, whether frontier models — that is, top-tier models — quietly get worse right after launch. It pilot-runs 2336 test items split into four test groups, four runs each, keeps the top 93 percent answered correctly every time, leaves only 78 items with an ambiguous verdict, and then settles the verdict through a preregistered procedure. Every day it runs 90 samples with harness hash 461391b6fce64167, command-line version 2.1.280, and frozen prompts to estimate drift statistically. Grading is built on Inspect, the inspection framework of the UK AI Security Institute, and the statistical treatment follows Anthropic's Adding Error Bars to Evals, the error-bar paper on large language model evaluation. The eight exams and the calibration scripts ship together, so the panel composition and the grading can be reproduced, and the preregistration documents (PREREGISTRATION.md 26KB, PLAN.md 21KB) are kept separate from the result-calibration documents (CALIBRATION.md, VALIDATION.md, DESIGN.md).
That said, as of October 17, 2026 the record covers only 25 of 30 days, and the minimum detectable effect computed over a 10-day baseline window is 6.6 points per 10-day window. Changes smaller than 6.6 points per 10-day window are hard to detect with this design. The implementation is small in scale, and the benchmark is tied to a single model from a single vendor, which also limits how far the results generalize. In the collected evaluation scores, momentum is low at 3.7 on a 10-point scale, while novelty is 8.0, technology 7.0, and quality 8.0. The record is still thin, so what can be expected right now is the drift-monitoring methodology itself.