Local Inference Engines
Standalone engines and libraries that execute trained models on a user's own CPU, GPU, NPU, or Apple Silicon. Apply the complete-engine precedence defined by llm-inference first: production or distributed serving and constrained-device runtimes belong elsewhere; modality-specialized engines belong under multimodal-runtime-inference unless Apple-specific or other hardware-specific specialization takes precedence. Excludes model artifacts, training tools, runtime-management utilities, and hardware-fit calculators.
Subfields
Apple Silicon Inference
Complete local model execution engines primarily specialized for Apple Silicon, MLX, Metal, or Apple unified memory. Excludes standalone kernels, production serving systems, and runtimes primarily designed for constrained mobile or embedded devices; those take precedence even on Apple hardware.
3 projectsGeneral Local Engines
Portable inference engines designed to run many model architectures and quantization formats on local hardware; it excludes forks dedicated to one GPU vendor and desktop model launchers.
8 projectsHardware-Specific Engines
Complete local inference engines or performance forks primarily dedicated to a particular non-Apple accelerator or hardware architecture. Excludes standalone kernels, production serving, constrained-device runtimes, Apple-specific engines, and portable engines that merely support the hardware.
11 projects| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| ollama/ollama | A general-purpose inference engine that runs many open models locally or in the cloud. | ★ 183.7K | Go | MIT | v0.40.6 |
| ggml-org/llama.cpp | A portable C/C++ inference engine executes large language models locally. | ★ 131.6K | C++ | MIT | b11719 |
| niko1221/strata | A local engine for Qwen3.8-Flash-Next with one-click setup and OpenAI- or Anthropic-compatible localhost APIs. | ★ 27.2K | C++ | MIT | v0.1.45 |
| antirez/ds4 | Runs selected DeepSeek, GLM, and Qwen models locally across Metal, CUDA, and ROCm with SSD streaming. | ★ 23.9K | C | MIT | — |
| jundot/omlx | OMLX is a local Apple Silicon LLM server with continuous batching, SSD and KV caching, and menu-bar management. | ★ 22.9K | Python | Apache-2.0 | v0.7.1rc1 |
| flashml-org/freetoken | An engine that serves massive MoE models locally across GPU, CPU, and host memory. | ★ 14.6K | Python | Apache-2.0 | v0.1.3 |
| magnitudedev/magnitude | A general inference engine that autotunes kernels to run local models across CPUs and multiple GPU vendors. | ★ 7K | Rust | Apache-2.0 | @magnitudedev/cli@0.2.9 |
| syv-ai/hyperqwen | Patches and optimizes vLLM to serve large Qwen models quickly on a single 24 GB CUDA GPU. | ★ 2.1K | Python | Apache-2.0 | — |
| fesens/opentpu | A hardware accelerator stack that runs Qwen models on a Kintex-7 PCIe card. | ★ 766 | Verilog | Apache-2.0 | — |
| warpfront/hipfire | A complete Rust LLM inference engine for AMD RDNA GPUs with native HIP/ROCm kernels and an OpenAI-compatible server. | ★ 692 | Rust | NOASSERTION | v0.4.2.1 |
| lithos-ai/lithos-metal | A complete local LLM engine with Metal megakernels and speculative decoding for Apple Silicon. | ★ 290 | Python | Apache-2.0 | v0.1.2 |
| indras-mirror/xolotl | Runs very large MoE language models on one RTX 4090 by tiering experts across VRAM, system RAM, and NVMe. | ★ 81 | C++ | MIT | — |
| jesdga95/basalt | A local inference engine for running Qwen MoE models on NVIDIA Blackwell workstations. | ★ 72 | C++ | MIT | — |
| professorpalmer/bonsai-ada-surgery | A PrismML llama.cpp fork and RTX 4070 recipe for serving Bonsai 2 27B at 262K context with q8_0 KV. | ★ 70 | Python | NOASSERTION | bundle-20261017b |
| lateos-ai/reflex | A Rust inference engine that runs GGUF models on CUDA for real-time agent decision loops. | ★ 62 | Rust | MIT | v0.2.9-runpod-hub |
| shawnshekari/strixite | A HIP-based local inference engine purpose-built for AMD Strix Halo gfx1151 and Qwen3.8-Flash-Next. | ★ 60 | C++ | AGPL-3.0 | v0.3.1 |
| 1picassoai/coachwhip | An offline Apple-silicon MoE inference engine runs large models locally and exposes an OpenAI-compatible API. | ★ 58 | Metal | Apache-2.0 | v0.5.3 |
| jitllm/jitllm | A Go inference engine that JIT-generates CPU/GPU kernels and runs models larger than memory across multiple backends. | ★ 48 | Go | Apache-2.0 | v0.2.6 |
| valeriodolci/capy-flash | Runs Qwen MoE models on dual RTX 5070 Ti GPUs using NVFP4, tensor parallelism, and CPU offload. | ★ 44 | C++ | Apache-2.0 | v0.5.4 |
| seuros/mdma | A complete local GGUF inference engine spanning CPU, ROCm, Vulkan, CUDA, Metal, Level Zero, and Edge TPU backends. | ★ 14 | Rust | BSD-2-Clause | — |
| hardenedlinux/sonicboom | A native C++/CUDA tensor runtime for locally executing quantized GGUF models with a stable C ABI. | ★ 5 | C++ | GPL-3.0 | — |
| cosmonomic/pulsarlm | An LLM engine that drops old low-attention tokens to keep persistent chatbot and agent contexts small. | ★ 4 | C++ | Apache-2.0 | — |