microsoft/pyrit is an open source framework built to proactively identify risks in generative AI systems. It is a red teaming tool for security professionals and engineers, and the name PyRIT stands for Python Risk Identification Tool for generative AI. The most recent release is v1.1.0, and the changelog for this release is not presented in the confirmed materials.
The center of this framework is bundling attack generation, transformation, target execution, and scoring into a single procedure. It builds attacks against a target, transforms them to fit the goal, actually executes the target, and then scores the results to carry out repeatable adversarial testing. Because the same procedure can be replayed and tested, it serves as a basis for generative AI safety evaluation.
The license is MIT, the implementation language is Python, the repository was created on December 12, 2023, and the star count is 4663. The README points to a website with usage, installation, and contribution guidance, the Microsoft Security Response Center for reporting vulnerabilities privately, and a Discord server where the team and community can be reached.
That said, the points where the evidence runs out are also clear. From the materials of this repository alone, you cannot compare how much risk it catches on which models, and no benchmarks or verified figures are presented. Of the evaluation items provided, only Momentum was given a score of 3.5 out of 10, and the remaining items carry no numbers. It is usable for testing with a specific target and evaluation criteria set in place, but there is not yet a basis for treating it as a general-purpose risk assessment tool.