A commit f3cc3ec9 landed in the luce-org/lucebox repository on November 6, 2026. Its title is perf(qwen4exp): fewer target launches in split decode(#937), and it is a performance change that reduces the number of kernel launches the target GPU had been issuing every layer in the Qwen-family path that decodes across two GPUs. The change spans 18 files, with 993 lines added and 61 lines deleted.
lucebox is a server that runs LLM speculative inference on heterogeneous hardware and consumer GPUs. It first produces a short draft, and a larger model verifies that result; the direction of this project is to use devices of differing performance together within a single graph. The core of this commit is a feature that defers the join of one layer's routed expert sums to the next layer. The target GPU runs the next layer's attention and router, while the expert GPU runs the routed expert computation, overlapping them. Previously the two cards alternated, idling for roughly one third of the decoding time each, as the description notes. However, because this path changes the model's arithmetic, it is off by default and must be enabled directly with LUCE_QWEN4EXP_LATE_EXPERTS=all or <a>-<b>.
Fusions on the CUDA·HIP backend side increased as well. The router's GET_ROWS·SUM_ROWS·CLAMP·DIV were combined into a single execution, the shared expert's gate sigmoid and multiplication were folded into the MoE combine kernel, and q8_1 quantized rows were made readable directly by the next matmul, eliminating the separate quantization execution. The commit message states that, based on the decoding profile, the per-layer kernel count on the target GPU drops from 45.4 to about 36, and that decoding in the late-expert split becomes 1.5~2.5% faster. For AMD RDNA4 (gfx1151), a short-row Q8_0 MMVQ kernel was added, said to be 1.5~3 times faster in the 1~8 column range, and greedy decoding was changed to compute argmax on the GPU so that only token IDs, rather than the entire logit row, are read.
All of these numbers are values stated by the commit message, not independent measurements. Because the late-expert path changes arithmetic, leaving it on can produce output different from before, and the fusions also remain behind on/off flags. In exchange, a new test_qn_split_fusions test was added that checks whether the fusions actually apply and the argmax tie-breaking rule, and it was incorporated into CI. The right way to read this commit is as one that leaves hardware-specific, hand-tuned optimizations in the form of code and tests; how much the default configuration's performance changes is still too early to judge.