apple-aiml-research/ml-rsi-inference-bench is a long-horizon benchmark for coding agents that improve the inference stack of large language models. It is a repository created by the apple-aiml-research organization on October 16, 2026; the implementation language is Python, and there is no release tag or version notation. The repository has 105 stars.
The benchmark evaluates coding agents that target self-improving inference systems. Agents perform tasks using the actual vLLM or SGLang engine and GPUs. The tasks divide the model lifecycle into three stages: Day-0 Model Support is new model support, Inference Optimization is optimization during operation, and Bug Fix is regression fixes. Scores are assigned by whether the first stage passes and by the speedup on the remaining two tasks, and runs are carried out through the Pier framework. In other words, it measures the effect of the changes the agent applied to the engine rather than the model itself. Since the score is not the agent's self-report but is directly built and measured by a separate verifier, it is worth referencing from an agent evaluation design perspective. Public evaluations targeting self-improving agents have always been scarce, and cases open-sourced as official code are rare.
However, the unconfirmed points are clear as well. No result figures have been published yet and the paper is only planned, so it is difficult to gauge at present what improvements this benchmark actually produces. The license is also unconfirmed, so the terms of use are unknown. Those looking to compare measured performance would be better off waiting until the results are published.