Model Evaluation
Benchmarks, datasets, judges, and harnesses primarily measuring general model or agent capability, quality, reliability, or efficiency. Safety-, jailbreak-, injection-, harmfulness-, or security-focused evaluation belongs under ai-safety-evaluation regardless of benchmark or harness packaging. Excludes training tools and production execution systems.
Subfields
Evaluation Harnesses
Frameworks that execute repeatable evaluations, score traces, compare configurations, or build evaluation environments; it excludes standalone model weights and production agent runtimes.
1 projectsModel and Agent Benchmarks
Benchmark suites, leaderboards, datasets, and challenge environments for comparing models or agents; it excludes evaluation orchestration software that only runs externally supplied tests.
3 projects| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| scaleapi/agentenv-framework | A framework for researchers, engineers, and domain experts to jointly build realistic, reproducible RL environments. | ★ 322 | Python | Apache-2.0 | v0.9.1334 |
| qwenlm/blenderworld | 223-task Blender benchmark with 2,651 checks evaluates agents creating editable 3D scenes from text, images and video. | ★ 124 | Python | MIT | — |
| composo-ai/dishwasherbench | Benchmarks language-model agents on goal completion in a physics simulation of a real dishwasher. | ★ 70 | TypeScript | MIT | — |
| cmu-l3/reuf2f | A benchmark where coding agents prove theorems or counterexamples in Lean-formalized undergraduate research problems. | ★ 9 | Lean | NOASSERTION | — |