New
srkrz23/vramkit — two-track VRAM compression for weights and the KV cache
srkrz23/vramkit is a Python repository implementing a lossless weight codec and KV cache compression that applies different quantization axes to keys and values.
CURATED OPEN SOURCE INTELLIGENCE
Inference & serving
New
l-forster/open-jet v0.5.0 — Offline Agent and CUDA Inference Engine
l-forster/open-jet provides an offline coding agent for local models and a CUDA inference engine for Qwen3.8-27B; the latest release is v0.5.0.
New
indras-mirror/xolotl — a large MoE inference engine for a single GPU
Xolotl, a C++ engine that extends Strata, serves several large models through an OpenAI/Anthropic-compatible server, and its README reports a decode speed of 26.8 tok/s for DeepSeek-V4.1-Flash 552B on an RTX 4090.
New
nandhakishorm/laya v0.4.9 — non-autoregressive decision engine
A look at v0.4.9 of nandhakishorm/laya, the scope of its engine that performs selection, scoring, and yes/no decisions, and the limitations it publicly states.
Release
niko1221/strata v0.1.45 — Qwen3.6 image input and concurrent requests
niko1221/strata v0.1.45 adds image input support for Qwen3.6-35B-A3B and handles up to 8 concurrent requests, with performance gains that vary depending on available VRAM.
New
alexsjones/llmfit v1.1.17 — Model recommendations by hardware
alexsjones/llmfit v1.1.17 is a Rust command-line tool that recommends language models suited to your hardware and shares measured benchmarks.
New
antirez/ds4 — a local inference engine for Metal, CUDA, ROCm
antirez/ds4 is a C-based local inference engine that runs a set of models on Metal, CUDA, and ROCm.
New
magnitudedev/magnitude — a local inference engine tailored to the hardware
magnitudedev/magnitude is a Rust-based open-source engine that tunes kernels to the hardware and uses them for model inference.
New
flashml-org/freetoken v0.1.3 — a MoE serving engine for desktops
An introduction to what flashml-org/freetoken offers for serving large MoE models on edge hardware, and to the limits of its performance information.
Release
Niko1221/Strata v0.1.44.1 — Legacy CPU SIGILL hotfix
Strata v0.1.44.1 fixes SIGILL errors on older CPUs, build problems, and the server's crash retry behavior.
Release
ggml-org/llama.cpp b11630 — Daily Build Automated Release
ggml-org/llama.cpp b11630 is a daily build automated release whose changes are described only at the level of commit messages.
No matching stories.