github.devotd.
ReleaseInference & serving

professorpalmer/bonsai-ada-surgery bundle-20261018d — Slot Cache Save Path Change

Covers the change in bundle-20261018d, which keeps the engine as-is and switches the slot cache save path, along with the latency results the repository reports.

Repositoryprofessorpalmer/bonsai-ada-surgery ↗

Bonsai 2 27B: 262k context (trained max) on 12 GB, 54 to 97 tok/s decode, 2x prefill, q8_0 KV recipe, grammar tool calls 9/9. Same ternary weights. Windows bundle + receipts.

stars
★ 72
Forks
6
Language
Python
License
NOASSERTION
Latest release
bundle-20261018e
Read original source ↗View on GitHub ↗

bundle-20261018d of professorpalmer/bonsai-ada-surgery is a Windows release bundle that touches the slot cache save path without changing the engine. According to the repository description, the project deals with Bonsai 2 27B. Instead of rewriting the entire conversation, the release now records only the tokens after the last save as a diff. The source is the repository's public bundle-20261018d page.

This change applies to the task of saving the conversation, not to the model inference engine. So the point of the update is not how answers are generated but the change in save path. The repository reports that average answer wait time in a 30-turn session on an RTX 4070 with the 8GB preset dropped from 53.2 seconds to 14.2 seconds. It states that all 30 responses were identical. What this release presents is that in that environment and session, average wait time fell while response content was preserved.

The release bundle is 616,681,355 bytes with a SHA-256 checksum provided alongside, and the commit number is 191. The executable is byte-for-byte identical to the previous bundle, and the engine commit is the same too. This shows that the deployment did not change the executable or the engine but the save path. That said, the latency figures are limited to the 30-turn result the repository gives for an RTX 4070 with the 8GB preset. There is no evidence that the same reduction appears on other devices or in other sessions, so it is hard to read it as a general performance improvement.

More in this field

Local Inference Engines ↗