confident-ai/deepeval is an open source framework for testing the quality of large language model (LLM) applications, and its most recent release is python-v4.2.4. The repository describes itself as an LLM evaluation framework, is written in Python, and is distributed under the Apache-2.0 license. It has 18993 GitHub stars, and the repository was created on August 10, 2023.
The README explains that this project is a tool specialized in unit testing LLM apps like Pytest, that is, checking one by one whether each feature works as expected. It provides metrics such as G-Eval, task completion, answer relevancy, and hallucination to evaluate an app as a whole, and it looks separately at retrieval-augmented generation (RAG), which searches for external knowledge and combines it into answers, at the trajectory, the sequence of actions of an AI agent, at the individual steps within it, and at conversation quality. It integrates with several frameworks including LangChain and OpenAI, and it states that evaluations can be run against traces from production environments in Confident AI.
However, the provided records do not include what python-v4.2.4 changed, so that cannot be confirmed. The momentum score left in the records is 5.9 out of 10, and no scores were given for novelty, technology, or quality. The list of metrics and the scope of integrations are transcribed from the README description, so they are not results demonstrating actual evaluation accuracy. Both the release changes and the measured performance of the metrics remain unverified.