giskard-ai/giskard-oss is an open-source Python library for testing and evaluating LLM (large language model) agents. The latest release in this repository is giskard-agents/v1.0.5, and the source is the giskard-ai/giskard-oss repository on GitHub.
It is licensed under Apache-2.0 and has 5913 GitHub stars. It was created on March 6, 2022, and the README describes the library along two tracks. Giskard Checks uses either scenarios you build yourself through the scenario API or built-in checks, and validates nondeterministic outputs with LLM-as-judge (a method where the model judges the results itself).
Giskard Scan handles red-teaming scenarios (testing by imitating an attacker), vulnerability scanning (testing known weakness types to find safety issues), and RAG (answering on the basis of retrieved documents) quality assessment. Using multi-turn tests, where a conversation continues over several exchanges, to pinpoint where a real agent gets blocked by an attack is the direction this project aims for.
That said, there are limits worth checking. What changed in giskard-agents/v1.0.5 is not in the given material, and no benchmark results demonstrating performance were provided either. The only observed momentum metric is low, 1.8 out of 10, and the remaining evaluation metrics are absent from the material. Until the release changelog is confirmed, it is more realistic to rely only on the feature scope described in the README.