marco-lu2508/fllm is a local LLM inference engine written in Rust alone. It reads GGUF models and provides OpenAI-, Anthropic-, and Ollama-compatible APIs. The 2026-11-07 commit fbf33f5 added memory prefetch (prefetch) to kernel_extra, the shared kernel used by the Q2_K, Q3_K, and IQ4_XS quantization formats, which bundle weights together and compress them for storage at roughly the 2–4 bit level. The integer arithmetic is unchanged; only the timing of reading blocks in from memory changes.
This prefetch is the approach the Q4_K–Q6_K kernels were already using. It requests four blocks ahead of each row (row) in advance, but the shared kernel left that to the hardware prefetcher. The reason is that each row flows as its own separate memory stream only a few blocks long, so the hardware prefetcher could not get its prefetch going. This time the same request was inserted ahead of each block, and it was applied not only to single-row decoding but also to the kernels used for the verification pass (verification pass). The measurements recorded in the commit message are the median of three alternating runs of the two builds, under conditions of 320MB of weights, one row, a release build, and 2 threads. Q2_K fell from 28.3ms to 19.1ms, a 32% reduction, and IQ4_XS fell from 15.5ms to 13.4ms, a 13% reduction. Q3_K showed no change, from 19.1ms to 18.9ms (run-to-run variation of 18.3–21.4ms), the control Q4_K stayed at 10.8ms, and Q6_K went from 11.0ms to 11.6ms.
That said, these figures are at the kernel level. Because there was no actual model, Q2_K and IQ4_XS were measured on the kernels alone with random weights, and end-to-end performance was not confirmed. The cpu_quantized, projection_kernels, the_exact_choice, and noise_of_a_verification_pass tests, which check whether values are preserved exactly, all passed. The reason Q3_K did not improve is that the limit lies not in the memory requests but in the decoding of 3-bit values and 6-bit scales, and that remains the task to look at next. The prefetch approach is one whose benefit has been confirmed for Q2_K and IQ4_XS, but it is too early to judge whether the same magnitude will come out across a whole model.