waldiez/slm-callsite-eval is the repository containing the benchmark, evaluation harness, and score records for the NeurIPS 2026 SLM-Agents workshop paper 'Not Every Call Needs a Frontier Model'. The commit 4bc44f2 of October 14, 2026 touched only one file, README.md, and no source code. It added 18 lines and deleted 1 line, and the content consists solely of links and an example. An arXiv link (2610.09021), a Hugging Face paper page, a Hugging Face dataset URL, and a badge were added, together with a Python code snippet that opens a subset of the data using `load_dataset`. The commit URL is https://github.com/waldiez/slm-callsite-eval/commit/4bc44f2e42e493ac88085bb13e2f36d24d7abf26.
What this repository measures is the open-source home automation multi-agent framework Wactorz. It fixes the five points where the agent calls the model (call sites), and scores nine models — combining local and hosted ones — separately for each call site. The README states that there are 280 benchmark cases across the five call sites, and the dataset now published on the Hub splits the data into per-call-site subsets and also contains the scored model outputs and the end-to-end evaluation results. You can read the test cases and the model outputs with the `datasets` library without downloading the repository.
That said, this commit changed only the documentation. The harness, the cases, and the scoring method are unchanged, and there are no newly evaluated models and no new scores. The README keeps a 'Joint' metric that harmonically averages execution accuracy and correct refusal, and separately measures 'Silent', the ratio of executable requests answered with an empty action list. The 39 impossible requests are included in that scope. Since neither the commit nor the repository description states which model produced which value, you have to check the arXiv original text and the Hugging Face dataset directly in order to compare the figures. Which model is superior cannot yet be judged from this repository alone.