The latest release of professorpalmer/bonsai-ada-surgery, bundle-20261017b, contains a recipe and measurement log for serving Bonsai 2 27B on a 12GB RTX 4070. The repository covers running it on a PrismML fork of llama.cpp without requantizing the 1.58-bit ternary quantized weights. With a q8_0 KV cache, the model's maximum trained context of 262,144 tokens is maintained.
The core piece is a hierarchical KV cache. KV stays in VRAM up to roughly 119k positions, after which system RAM takes over. In the README's single-slot measurement table (2026-10-08, RTX 4070 12GB), decode drops from 102 tok/s at 4k to 8.6 tok/s at 258k, and prefill drops from 1,173 tok/s to 436 tok/s. It also records an optimization that replaces tool schema grammar copying with binary search. It reports that with 107 tools, throughput at 63k rises from 41.8 tok/s to 61.2 tok/s.
That said, all the presented performance numbers and benchmarks are self-measured on a single RTX 4070. Independent reproduction depends on the Windows bundle and scripts, and the license is marked NOASSERTION, so reuse conditions are not given. This record should therefore be read as a serving result for that specific environment, and it does not guarantee performance in other environments.