The September 24, 2026 commit bfd685e in simreal-ai/simreal-mlbench is a documentation change that marks this repository as a public preview. It adds a status badge to the top of the README, along with a notice stating that this repository is the public, minimal component of the ML Benchmark and that the evaluation runtime and official scoring path, as well as the judging of the full suite, run in the SimReal engine and are therefore not present in this repository. A paragraph is appended at the end of the document stating that this repository is one of the public previews of the SimReal product family, with the full list of previews linking to github.com/Simreal-AI. Only the SHA256 hash of the changed README.md was updated in SHA256SUMS, and with 11 lines added and 1 line deleted, the code and the scoring method are unchanged.
This repository is an external scoring evaluation environment for machine learning research agents. The README states that 60 tasks span representation learning, prediction, vision, language, speech, multimodal learning, and structured scientific data. Agents are to understand datasets and develop modeling approaches, run experiments, and then submit predictions, and the evaluation targets are defined as the entire research workflow: data understanding, feature engineering, validation design, model selection, hyperparameter tuning, ensembling, debugging, resource allocation, and robust delivery. The published fragment contains the 60-task catalog, the evaluation protocol, the scoring code, and the client SDK, and the README notes that this repository drew inspiration from openai/mle-bench, but that task selection, the resource protocol, and the scoring method differ. Usage instructions are split out into a separate document describing the agent guide, environment setup, gateway, and release structure.
The problem is that official scores cannot be directly reproduced. The evaluation runtime and official scoring path, as well as the judging of the full suite, are in the SimReal engine, and this repository has no execution path. The README states that it is at an early stage and will change, and directs those seeking full access for labs and partners to contact business@simreal.co. The license type is not given in this commit record. Since this commit was limited to a documentation change, it is too early to judge changes in scores or task composition, and if one plans to attach an agent, it is worth considering first that the scoring path is closed.