magnitudedev/magnitude is an open-source local inference engine that tunes its kernels for the specific computer when a model is downloaded. A kernel is the code that performs model computations on a GPU or CPU. The project is written in Rust and is published under the Apache-2.0 license. It says it supports Apple Silicon, NVIDIA, AMD and CPUs.
The core of Magnitude is finding the execution configuration that fits the hardware and using it for model inference. The README says it is on average more than 2x faster than Ollama, and cites, based on 11 popular local models, 1.32x prefill and 2.19x decode on Mac, and 1.34x prefill and 2.17x decode on NVIDIA. Prefill is the stage that processes the input context, and decoding is the stage that keeps generating tokens after that. During tuning it varies the GPU thread configuration, chunk size and data prefetch distance, selecting the combinations that leave the output unchanged. According to the README, tuning takes up to 1 minute.
It additionally provides per-model speculative decoding drafters and a quantized KV cache. The KV cache halves per-agent memory and returns the memory once the agent finishes. It also shares the prefix cache across sessions. A desktop app that bundles a CLI can connect to Pi, OpenCode, Hermes and Codex. That said, the speed figures are results presented by the README, and the provided materials alone do not confirm whether they can be independently reproduced. So the comparison figures are best read as the performance Magnitude itself claims.