Hardware-Specific Engines
Complete local inference engines or performance forks primarily dedicated to a particular non-Apple accelerator or hardware architecture. Excludes standalone kernels, production serving, constrained-device runtimes, Apple-specific engines, and portable engines that merely support the hardware.
| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| syv-ai/hyperqwen | Patches and optimizes vLLM to serve large Qwen models quickly on a single 24 GB CUDA GPU. | ★ 2.1K | Python | Apache-2.0 | — |
| fesens/opentpu | A hardware accelerator stack that runs Qwen models on a Kintex-7 PCIe card. | ★ 768 | Verilog | Apache-2.0 | — |
| warpfront/hipfire | A complete Rust LLM inference engine for AMD RDNA GPUs with native HIP/ROCm kernels and an OpenAI-compatible server. | ★ 692 | Rust | NOASSERTION | v0.4.2.1 |
| indras-mirror/xolotl | Runs very large MoE language models on one RTX 4090 by tiering experts across VRAM, system RAM, and NVMe. | ★ 88 | C++ | MIT | — |
| jesdga95/basalt | A local inference engine for running Qwen MoE models on NVIDIA Blackwell workstations. | ★ 72 | C++ | MIT | — |
| professorpalmer/bonsai-ada-surgery | A PrismML llama.cpp fork and RTX 4070 recipe for serving Bonsai 2 27B at 262K context with q8_0 KV. | ★ 70 | Python | NOASSERTION | bundle-20261018 |
| lateos-ai/reflex | A Rust inference engine that runs GGUF models on CUDA for real-time agent decision loops. | ★ 62 | Rust | MIT | v0.2.9-runpod-hub |
| shawnshekari/strixite | A HIP-based local inference engine purpose-built for AMD Strix Halo gfx1151 and Qwen3.8-Flash-Next. | ★ 60 | C++ | AGPL-3.0 | v0.3.1 |
| valeriodolci/capy-flash | Runs Qwen MoE models on dual RTX 5070 Ti GPUs using NVFP4, tensor parallelism, and CPU offload. | ★ 44 | C++ | Apache-2.0 | v0.5.4 |
| hardenedlinux/sonicboom | A native C++/CUDA tensor runtime for locally executing quantized GGUF models with a stable C ABI. | ★ 5 | C++ | GPL-3.0 | — |
| cosmonomic/pulsarlm | An LLM engine that drops old low-attention tokens to keep persistent chatbot and agent contexts small. | ★ 4 | C++ | Apache-2.0 | — |