SWE-bench is a benchmark that puts real software issues collected from GitHub in front of language models. The repository swe-bench/swe-bench provides a setup where, given a codebase and an issue, the model generates a patch that resolves the problem, together with a reproducible evaluation method using Docker and guidance on using the datasets. The README frames the benchmark's question as whether language models can solve real GitHub issues. No release version is listed in the material.
What sets this benchmark apart is that it evaluates issues collected from real repositories as they are. Because the codebase is kept intact and reproducible tests verify whether the patch solves the problem, it comes closer to measuring a model's ability to read a repository's structure and carry a fix through. The evaluation being reproducible in a Docker environment means the results can be checked again elsewhere. The implementation language is Python and the license is MIT. The repository was created on October 4, 2023, and has 6061 stars.
That said, this material contains no measured results for any specific model. The only scored figure available is a momentum score of 3.9, and no novelty (신규성), tech (기술), or quality (품질) scores are given. The star count and the momentum score are values recorded at a certain point in time, so they do not stand in for performance. The evaluation tooling and datasets are in place, but there is no comparison showing which model reaches which level, so for now it is closer to a tool worth trying out directly.