github.devotd.
NewEvals

intel/dokimi — Measuring the Effect of Coding Agent Artifacts

intel/dokimi is a Python framework that uses paired comparison to measure whether agent artifacts such as skills and documents actually help the model.

Repositoryintel/dokimi ↗

Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.

stars
★ 7
Forks
6
Language
Python
License
Apache-2.0
Read original source ↗View on GitHub ↗

intel/dokimi is a Python framework that measures whether artifacts supplied to coding agents — skills, MCP document sources, agent profiles, or repository instructions — actually help. According to the repository description, it compares on and off conditions, analyzes new against already-known content, and scans the repository through graded agent runs. In the paired comparison it changes a single element, fixes everything else, and runs the same set of items. It reports the change against the baseline together with a significance level.

The focus of the tool is measurement results, not the popularity of artifacts. The README contrasts publishing results as scorecards with a popularity-ranking marketplace. It uses whichever models can be connected through LiteLLM and is not limited to a specific vendor. The statistics code is split into binary statistics and rank correlation analysis modules, each with its own test file. An Intel copyright notice and an Apache-2.0 license are present.

That said, the repository was created on 2026-10-08, has 6 stars, and has no release. The code and test files are grounds for checking the computational implementation, but they do not prove the validity of the measurement design. No benchmark against other tools is presented either. So rather than treating it as a criterion for judging agent performance, it is better suited for examining an evaluation design that tries to compare the effect of artifacts.