rzc156/vla-diag is a diagnostic tool that checks what a VLA benchmark's success rate reflects beyond model performance. It is not a new release but a tool the repository introduces, a Python library that is not tied to a specific simulator or model framework. According to the README it uses Python 3.9 or later and numpy, and it is published under the Apache-2.0 license. Instead of presenting a success-rate number alone, its aim is to take the question apart: whether the differences in evaluation results are real, what type the failures are, and whether there is any room for choice.
The features are split into five modules: stats, integrity, failures, headroom, and report. stats covers differences in results and the required sample size, while integrity checks the integrity of checkpoints, actions, and the canary. failures distinguishes failure types such as timeouts and motion errors, and headroom looks at the margin available in selection-based approaches. report handles reporting the results. In the README's LIBERO application table, it explains that across 1,598 decision points and 900 paired rounds, 96% of decision points left no room for choice, and even a perfect chooser reached only +0.22pp (p=0.93). A world-model failure-prediction case presents 3 models and 100 rounds where the in-distribution AUC of 0.25~0.53 was no better than random. The defaults use a 95% interval and 0.8 power, and when the sample is insufficient the tool reports it as not measurable instead of giving a point estimate.
The README also includes a case from GPU deployment verification where the between-device difference of 0.014 was smaller than the sampling noise of 0.025. It names the Wilson score interval and McNemar's paired test as its statistical methods, and CHECKS.md records 7 real incidents. The figures in the application table do come from the repository's own analysis, however, and no external reproduction is presented. The diagnostic methods and cases can therefore serve as a starting point for examining evaluation numbers, but they are hard to read as evidence that the same results would appear in other environments.