Devotd Github
NewEvals

rails/lemans v1.6.0 — Miniswen prompt caching and ruby_llm 2.1.1 adoption

The Ruby agent benchmark harness rails/lemans was bumped to 1.6.0 on 2026-11-03, bringing ruby_llm 2.1.1 and sweeping prompt caching changes to Miniswen.

Repositoryrails/lemans ↗

The Ruby agent benchmark harness rails/lemans was bumped to 1.6.0 on 2026-11-03, bringing ruby_llm 2.1.1 and sweeping prompt caching changes to Miniswen.

GitHub description: A Ruby harness for running agent benchmarks

stars
★ 82
Forks
5
Language
Ruby
License
MIT
Read original source ↗View on GitHub ↗

rails/lemans is a Ruby harness that runs agent benchmarks and collects the results. The same repository also contains a separate module called Miniswen, and the 2026-11-03 commit 'Bump 1.6.0' raised the version files of both modules together, aligning them from 1.5.2 to 1.6.0. The change touches only 3 files, with 4 lines added and 2 lines deleted, and a new 1.6.0 entry dated 2026-11-03 was added to the CHANGELOG.

All changes included in 1.6.0 are on the Miniswen side. ruby_llm was upgraded to 2.1.1 to use built-in caching, finish reasons, and reasoning replay, while the roughly 10-minute retry window, the RUBYLLM_DEBUG flag, and the JSON 3 allowance were kept as they were. After that, prompt_cache_key was sent to all OpenRouter models so that the cache is maintained even when a session is not bound to a particular provider ordering, and prompt caching was enabled for all models so that models that need caching, like Claude (including the Anthropic API), are cached, while models that were cached automatically before remain unaffected.

That said, all this commit leaves behind is a CHANGELOG entry and version constants. As of commit 335272ae006d02f0f6a5b4b51e707f99b79d10a8, no performance figures, benchmark results, or license information are given, so there is no way to guess how much caching actually reduces, and no score change can be confirmed from this material. Teams choosing an agent harness that need cache persistence and observability (debug flag, finish reasons, reasoning replay) would be well advised to look at 1.6.0 directly, but this is not yet the stage to decide on adoption on the basis of performance improvements.