openai/multilangbench is a test suite that evaluates language models in several languages on top of SQLite. It asks the model to write an SQLite 3.40.1 implementation itself, then scores the produced executable by comparing its results with real SQLite. Whether the model used the requested implementation language also counts toward the judgment. The material shipped with it is the SQLite documentation and the converted tests. No public release has been confirmed. The listed primary language is PLpgSQL, the license is Apache-2.0, and it has 12 stars. The repository was created on October 19, 2026.
The criterion of this evaluation is the behavioral compatibility of the implementation the model produced. A single test splits into an initial database, the queries the model runs, and validation queries that inspect the results. The model has to produce the result of running the queries from the given initial state, and the validation queries check whether that result is right. According to the README, about 24,000 of the roughly 51,000 original TCL tests were converted into this form, and the default evaluation uses 272 of them. The subset evaluation lowers the cost of the whole test set, and the full test set is kept aside so that high-scoring results can be checked in detail.
Still, the evaluation scope is narrow, limited to SQLite behavioral compatibility, and the default evaluation stops at a subset of the converted tests. The collected scores are novelty 8.0, technical 6.0, quality 6.0, and momentum 4.3. Because the default scores alone make it hard to judge the accuracy of an implementation, the detailed check has to be left to the full test set.