xlang-ai/osworld is a benchmark that evaluates multimodal agents performing open-ended tasks in real computer environments. The work was introduced at NeurIPS 2024, and the repository was first created on October 16, 2023. The latest release is v0.1.16, and the changes included in this release are not recorded in the public records.
How evaluation works is the core of this project. A multimodal agent is a program that reads several forms of information together while operating a computer, and OSWorld has the agent perform the requested task across multiple applications inside a virtual machine, then checks the final state once the task is done and turns that into an evaluation. The README explains how to use VMware and VirtualBox for desktop execution and how to run it with Docker-based virtual machines, and it also mentions platform support such as AWS and Azure. The implementation language is Python and the license is Apache-2.0.
That said, the public records alone do not settle the judgment. There is no changelog for v0.1.16, and no per-agent performance figures remain. The research record lists only a Momentum score of 2.1 out of 10, with no novelty or technical completeness scores recorded. A team looking to evaluate desktop automation agents may find the runtime setup useful, but what changed in this version has to be checked directly in the repository's release history.