evolvinglmms-lab/lmms-eval is a multimodal model evaluation tool that covers text, image, video and audio tasks in a single tool. The latest version by GitHub release is v0.7.3, and the changes included in this release were not given as release notes.
This tool runs multiple models on the same evaluation tasks and the same procedure so they can be compared. An evaluation harness means a comparison frame that runs multiple targets under identical conditions. The README states that measuring the real capability of large multimodal models is its purpose. It supports image, video and audio evaluation tasks and multiple model families, and it says models can also be evaluated through OpenAI-compatible API endpoints. It groups work with model adapters and a common task configuration. It aims to produce consistent results on the same models and benchmarks, and treats reproducibility, efficiency and trustworthiness as core principles. The repository's primary language is Python, it was created on March 7, 2024, and the star count is 4461.
However, in the checkable materials the license is listed as NOASSERTION, so the actual usage terms need to be confirmed by looking at the repository directly. The newness, technology and quality scores are empty, and only the momentum score came out low at 2.1. What can be judged is the scope and structure of what this tool covers. For work that requires comparing by swapping in different models, it looks like a tool worth keeping on the candidate list.