jwuthri-cloud/tracely-ai is a Python tool that evaluates AI agent execution traces collected in production and connects them to CI regression testing. It groups failures from stored execution traces and pins each failure case as a test that can replay it. If a test fails, it provides a flow that blocks the PR for that change. The provided material contains no latest version number or release changes, so what follows covers the features the repository describes.
According to the README, Tracely structures agents, conversations, and each step from traces received via OTLP. It shows cost and latency information alongside evaluation results, and can replay stored tool responses and model responses to review execution outcomes. The center of this tool is turning failures not into simple observation logs but into repeatable regression cases. It also presents configurations for deploying the API, workers, web interface, and storage services into your own environment. It is MIT licensed and written in Python.
This approach focuses on checking the quality of agent execution before and after a change. However, FACTS records no latest release, and no criteria for computing evaluation scores or comparison test results are presented either. So the regression testing and PR blocking flow can be regarded as a feature the repository describes, but the evidence that it works in real environments has not been confirmed. Approaches to handling agent quality as regression testing are still close to an early stage.