Model Inference
Reusable engines, serving systems, kernels, caches, and utilities executing trained models. Classify the main delivered component first: management utilities, memory mechanisms, or kernels rather than a bundled engine. For complete engines, precedence is production or distributed serving, constrained-device execution, Apple-specific local engines, other hardware-specific local engines, modality-focused engines, then portable local engines. Hardware support or multimodal support alone does not establish specialization. Excludes model artifacts, training tools, agent loops, and end-user authoring applications.
Subfields
Embedded Device Inference
Compact inference runtimes for microcontrollers, phones, consoles, and other constrained devices; it excludes full server clusters and general embedded agent frameworks without local model execution.
1 projectsInference Memory and Cache
Technologies that reduce or virtualize model weights, activations, attention state, or KV-cache memory during inference; it excludes lossy model quantization done before deployment and ordinary RAM management.
3 projectsLocal Inference Engines
Standalone engines and libraries that execute trained models on a user's own CPU, GPU, NPU, or Apple Silicon. Apply the complete-engine precedence defined by llm-inference first: production or distributed serving and constrained-device runtimes belong elsewhere; modality-specialized engines belong under multimodal-runtime-inference unless Apple-specific or other hardware-specific specialization takes precedence. Excludes model artifacts, training tools, runtime-management utilities, and hardware-fit calculators.
22 projectsModel Runtime Tooling
Utilities for selecting models that fit hardware, launching runtimes, monitoring use, and managing local model files; it excludes the inference engine itself and general LLM API routing.
2 projectsMultimodal Runtime Inference
Reusable execution engines primarily specialized for image, audio, video, vision-language, or combined multimodal models. Complete-engine deployment and hardware precedence defined by llm-inference applies first. Excludes model releases, training tools, end-user media workflows, and general engines with incidental multimodal support.
2 projects| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| ollama/ollama | A general-purpose inference engine that runs many open models locally or in the cloud. | ★ 183.7K | Go | MIT | v0.40.6 |
| ggml-org/llama.cpp | A portable C/C++ inference engine executes large language models locally. | ★ 131.6K | C++ | MIT | b11719 |
| alexsjones/llmfit | LLMFit checks local hardware and recommends quantized models that fit available CPU, memory, GPU, and VRAM. | ★ 38.1K | Rust | MIT | v1.1.17 |
| niko1221/strata | A local engine for Qwen3.8-Flash-Next with one-click setup and OpenAI- or Anthropic-compatible localhost APIs. | ★ 27.3K | C++ | MIT | v0.1.45 |
| antirez/ds4 | Runs selected DeepSeek, GLM, and Qwen models locally across Metal, CUDA, and ROCm with SSD streaming. | ★ 23.9K | C | MIT | — |
| jundot/omlx | OMLX is a local Apple Silicon LLM server with continuous batching, SSD and KV caching, and menu-bar management. | ★ 22.9K | Python | Apache-2.0 | v0.7.1rc1 |
| flashml-org/freetoken | An engine that serves massive MoE models locally across GPU, CPU, and host memory. | ★ 14.6K | Python | Apache-2.0 | v0.1.3 |
| magnitudedev/magnitude | A general inference engine that autotunes kernels to run local models across CPUs and multiple GPU vendors. | ★ 7K | Rust | Apache-2.0 | @magnitudedev/cli@0.2.9 |
| flashml-org/freevideo | A local inference engine efficiently runs the MiniMax H3 video model on consumer GPUs. | ★ 2.1K | Python | Apache-2.0 | v0.4.14 |
| syv-ai/hyperqwen | Patches and optimizes vLLM to serve large Qwen models quickly on a single 24 GB CUDA GPU. | ★ 2.1K | Python | Apache-2.0 | — |
| kvmem/kvmem | KVMem virtualizes KV memory across llama.cpp and NInfer, offloading cache state between host memory and GPU. | ★ 1.1K | C++ | — | v0.18.0 |
| receptron/laya | Runs the Laya ONNX decision model in Node.js to produce calibrated classification probabilities and scores. | ★ 960 | TypeScript | MIT | v0.2.0 |
| fesens/opentpu | A hardware accelerator stack that runs Qwen models on a Kintex-7 PCIe card. | ★ 768 | Verilog | Apache-2.0 | — |
| warpfront/hipfire | A complete Rust LLM inference engine for AMD RDNA GPUs with native HIP/ROCm kernels and an OpenAI-compatible server. | ★ 692 | Rust | NOASSERTION | v0.4.2.1 |
| lithos-ai/lithos-metal | A complete local LLM engine with Metal megakernels and speculative decoding for Apple Silicon. | ★ 290 | Python | Apache-2.0 | v0.1.2 |
| surya-koritala/glyd | A lossless runtime codec for BF16 weights and KV-cache data that reduces GPU memory and reconstructs exact values. | ★ 255 | Rust | NOASSERTION | v0.32.1 |
| nyosegawa/speech.cpp | Runs speech synthesis, recognition, and VAD models across multiple hardware backends using ggml. | ★ 180 | C++ | MIT | v0.8.10 |
| indras-mirror/xolotl | Runs very large MoE language models on one RTX 4090 by tiering experts across VRAM, system RAM, and NVMe. | ★ 88 | C++ | MIT | — |
| jesdga95/basalt | A local inference engine for running Qwen MoE models on NVIDIA Blackwell workstations. | ★ 72 | C++ | MIT | — |
| professorpalmer/bonsai-ada-surgery | A PrismML llama.cpp fork and RTX 4070 recipe for serving Bonsai 2 27B at 262K context with q8_0 KV. | ★ 70 | Python | NOASSERTION | bundle-20261018 |
| lateos-ai/reflex | A Rust inference engine that runs GGUF models on CUDA for real-time agent decision loops. | ★ 62 | Rust | MIT | v0.2.9-runpod-hub |
| shawnshekari/strixite | A HIP-based local inference engine purpose-built for AMD Strix Halo gfx1151 and Qwen3.8-Flash-Next. | ★ 60 | C++ | AGPL-3.0 | v0.3.1 |
| 1picassoai/coachwhip | An offline Apple-silicon MoE inference engine runs large models locally and exposes an OpenAI-compatible API. | ★ 58 | Metal | Apache-2.0 | v0.5.3 |
| jitllm/jitllm | A Go inference engine that JIT-generates CPU/GPU kernels and runs models larger than memory across multiple backends. | ★ 48 | Go | Apache-2.0 | v0.2.6 |
| valeriodolci/capy-flash | Runs Qwen MoE models on dual RTX 5070 Ti GPUs using NVFP4, tensor parallelism, and CPU offload. | ★ 44 | C++ | Apache-2.0 | v0.5.4 |
| seuros/mdma | A complete local GGUF inference engine spanning CPU, ROCm, Vulkan, CUDA, Metal, Level Zero, and Edge TPU backends. | ★ 14 | Rust | BSD-2-Clause | — |
| andrei-borcea/liyab | Runs GGUF language models on phones by streaming models larger than RAM from flash and adapting to thermal conditions. | ★ 7 | C++ | Apache-2.0 | — |
| hardenedlinux/sonicboom | A native C++/CUDA tensor runtime for locally executing quantized GGUF models with a stable C ABI. | ★ 5 | C++ | GPL-3.0 | — |
| cosmonomic/pulsarlm | An LLM engine that drops old low-attention tokens to keep persistent chatbot and agent contexts small. | ★ 4 | C++ | Apache-2.0 | — |
| srkrz23/vramkit | Provides experimental compression for LLM inference weights and KV caches to reduce VRAM use. | ★ 2 | Python | — | — |