open-compass/vlmevalkit is an open-source evaluation toolkit that measures the performance of large vision-language models (LVLMs). Its targets are large multimodal models (LMMs) that handle both images and text, and the latest release of this repository is v0.2. The repository description states that it supports more than 220 LMMs and more than 80 benchmarks. The basic flow of evaluation is a generation-based approach: generate the model's answers and then match them against the ground truth.
The tool bundles multiple VLMs and multiple benchmarks into a single run-and-score pipeline, so that results for different targets can be compared under the same procedure. The README explains that you can evaluate across multiple benchmarks with a single command, and that it applies generation-based evaluation to all LVLMs, offering both an exact-match method and a method that uses an LLM to extract the correct answer from the model's response. The Python package is named vlmeval, and when evaluating your own model you only need to implement generate_inner(); the tool then handles data downloading and preprocessing, inference, and metric computation. It is implemented in Python and licensed under Apache-2.0, and it was created on December 1, 2023; it is a public repository with 4,448 stars.
That said, it is impossible to know what v0.2 changed, because the provided record contains no list of changes. The score items in the research record are on a 10-point scale, but only momentum received a score (2.1), and the novelty, technology, and quality items were left without values, so most other grounds for judgment are empty. The rationale item that holds the reasoning behind the judgment is also unfilled. There is no basis for describing the extent of the improvement, and the changes must be checked directly in the release notes. Looking only at the facts contained in the record, for now the realistic use is to run multiple VLMs and multiple benchmarks under the same procedure and compare the results.