The latest release of modelscope/evalscope is v1.14.0. EvalScope is a Python framework created by ModelScope Community, and its README explains that model capability evaluation and inference performance stress testing can be started with one command. The target scope is broad, spanning LLM, VLM, Embedding, Reranker, AIGC and agents.
Evaluation covers benchmarks such as MMLU, C-Eval and GSM8K, and it is integrated with several evaluation backends including OpenCompass, VLMEvalKit and RAGEval. On the performance side, it measures metrics such as TTFT (time to first token) and TPOT (time per output token), and a Web Dashboard allows model comparison, evaluation reports and detailed results. The repository was created in December 2023, the license is Apache-2.0, and it has 3565 stars.
However, what changed in v1.14.0 cannot be confirmed, because this record contains no release notes. The only grounds are the feature scope stated in the README and a momentum score of 2.8 (out of 10). No performance measurement figures are included — only a statement that it measures the metrics — so it is hard to judge which model is better. Bundling model capability evaluation and performance stress testing into one tool is a practical setup, but with the changes unknown, it looks like a reasonable scope to explore the features first.