openai/evals is a framework for evaluating large language models (LLMs) and LLM-based systems, and also a public evaluation registry. An eval means testing a model against a fixed set of items. The repository is written in Python, was created on January 23, 2023, and has 19,621 GitHub stars. The source for this post is the openai/evals repository on GitHub.
The repository has two axes. One is the public evaluation registry, which collects evals that test various aspects of OpenAI models; the other is a framework for running evals you author yourself. The registry is an openly published list of benchmarks, and on top of it you can implement and run custom evals tailored to your own data. The README notes that you can also build private evals that keep data unpublished and follow your business workflow. It explains that evals can be configured and run in the OpenAI Dashboard.
There are limits worth checking. There are no public releases, so the latest version number cannot be pinned down, and the license is listed as NOASSERTION, so reuse terms need a separate review. The observed momentum metric is low, at 2.1 out of 10. Running it in practice requires an OpenAI API key and Python 3.9 or later, so it is better to check the scope of the public evals and how private evals are implemented before deciding whether to adopt it.