jitllm/jitllm is an inference engine written in Go. It uses JIT compilation, generating kernels matched to the CPU or GPU when loading a model and then executing them. The latest release is v0.2.6, published under the Apache-2.0 license. The provided information contains no itemized changes for this version.
According to the README, it generates SSE, AVX2, AVX-512 and VNNI machine code as well as NEON on the CPU side, and PTX for CUDA, SPIR-V for Vulkan, Metal shaders and ROCm code for AMDGPU on the GPU side. It bundles disk, RAM and VRAM into a single paging layer to handle models larger than memory, and in mixture models it places each expert on its own page so that only the experts routed for a token are read. In the README's own benchmark, Qwen3-Next-80B prefill performance on 4 V100s was 4.62x that of llama.cpp, and among 35 models from 0.4B to 120B all of them surpassed llama.cpp in decode and 34 of them in prefill.
These figures come from the README's own benchmark results and do not mean an independent performance comparison. The README also notes that the ROCm path has not been tested on AMD hardware and that there is no interpretation-based fallback path. The performance comparison should therefore be read with the scope of validation in mind, and on AMD environments it is worth considering that ROCm support is hard to confirm.