github.devotd.
ReleaseInference & serving

shawnshekari/strixite v0.4.0 — 8-bit KV cache becomes the default

shawnshekari/strixite v0.4.0 makes the 8-bit KV cache the default and fixes problems with re-reading cached conversations in long chats and repeated decoding of the same image.

Repositoryshawnshekari/strixite ↗

LLM inference engine written from scratch in HIP for AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next - ~50 tokens/s out to 492k context

stars
★ 61
Forks
7
Language
C++
License
AGPL-3.0
Latest release
v0.4.0
Read original source ↗View on GitHub ↗

shawnshekari/strixite v0.4.0 is an update to the HIP-based inference engine for AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next. According to the GitHub release notes, this version promotes the 8-bit KV cache to the default. It also fixes a problem where the prompt cache re-read the conversation on every turn in long conversations, and a problem where the same image was decoded repeatedly.

The release notes state that at a 512k context the memory for one conversation state drops from 13.5 GiB to 8.3 GiB, and the GPU buffer shrinks from 95.3 GiB to 91.9 GiB. Conversations stored in the prompt cache become roughly 40% smaller. Decoding the same 20 images drops from 1.76 seconds to 0.07 seconds. Prefill is 3% faster, while decode throughput is described as unchanged within the margin of error. The repository description says it delivers about 50 tokens per second at up to 492k context.

These numbers and changes come from the v0.4.0 release notes on GitHub, and the provided information does not state the measurement environment or the impact on response quality. The throughput in the repository description is also given without measurement conditions. The memory and image decoding improvements can therefore be read as the results the release reports, but quality changes or performance in other environments are hard to judge from this information alone.

More in this field

Local Inference Engines ↗