github.devotd.
NewEvaluation Harnesses

scaleapi/agentenv-framework v0.9.1336 — Agent Evaluation Environment SDK

scaleapi/agentenv-framework is a Python SDK and CLI that builds and deploys agent evaluation environments as versioned images and grades the work agents perform.

Repositoryscaleapi/agentenv-framework ↗

A framework for researchers, engineers, and domain experts to jointly build realistic, reproducible RL environments.

GitHub description: Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.

stars
★ 349
Forks
30
Language
Python
License
Apache-2.0
Latest release
v0.9.1336
Read original source ↗View on GitHub ↗

scaleapi/agentenv-framework is a Python SDK and CLI that creates environments for agent evaluation and grades the work agents perform inside them. The latest version in the GitHub release history is v0.9.1336, and it follows the Apache-2.0 license. However, the change log for v0.9.1336 is not present in the given record.

An environment is defined as a container server that uses the public protocol agentenv-framework-protocol. It is built into a versioned image, deployed behind a gateway, and the deployed environment handles the grading of agent tasks. The bundle folder pins references to the versions held in the store, providing both local execution and editing. The deployment name is agentenv-framework, the import package is agent_env, and the command is agent-env.

The problem the project targets is building realistic reinforcement learning (RL) environments. The repository description states that researchers, engineers, and domain experts must work together along several axes such as artifacts, environment tooling, environment changes, and reproducibility. The repository argues that no open-source framework existed that could build such environments effectively. The part of the code that supports RL environment construction is the episode API. Only one or two agents are played directly in the outer learning loop, and the framework runs the remaining episodes. The star count shown on GitHub is 349.

However, there are no performance figures or first-party benchmark results, and the key feature, the Vercel sandbox provider, has not been released yet. The evaluation scores are novelty 7.0, technology 6.0, quality 8.0, and momentum 4.1, and the low technology score is due to the lack of quantitative performance evidence. It is premature to decide on adoption without performance evidence. It is better to judge once the sandbox provider and first-party benchmarks are out.