Devotd Github
NewInference & serving

rustingstudio/rustingstrata v1.2.0 — CPU·GPU Hybrid Inference Engine

rustingstudio/rustingstrata is a Rust inference engine that runs Qwen-family GGUF models, providing CPU·GPU hybrid processing and CUDA·Vulkan paths.

Repositoryrustingstudio/rustingstrata ↗

rustingstudio/rustingstrata is a Rust inference engine that runs Qwen-family GGUF models, providing CPU·GPU hybrid processing and CUDA·Vulkan paths.

GitHub description: Pure-Rust MoE inference engine for Qwen3.5/3.6/3.8 GGUF. CPU+GPU hybrid experts, NVRTC CUDA kernels, OpenAI-compatible server. 2x llama.cpp prefill and decode on a 12 GB RTX 3060.

stars
★ 93
Forks
13
Language
Rust
License
NOASSERTION
Latest release
v1.2.0
Read original source ↗View on GitHub ↗

The latest release of rustingstudio/rustingstrata is v1.2.0. This repository is a pure Rust inference engine that runs Qwen-family GGUF models, and the provided record lists no items newly added or changed in v1.2.0. The repository description states that prefill and decode performance are each twice that of llama.cpp on a 12 GB RTX 3060.

The engine provides a hybrid execution mode that processes on CPU and GPU together when the model's expert modules do not fit in VRAM. Expert modules are the parts of computation that are selected according to the input in Mixture-of-Experts (MoE) models. The execution paths support CUDA and Vulkan, and the CUDA kernels are compiled at runtime using NVRTC. According to the repository description, the inference server also supports an OpenAI-compatible chat API and tool calling. On Reddit r/Qwen_AI, a claim was shared that, based on Qwen3.8-Flash-Next, decoding is 2x faster and prefilling is 2.5x faster than the original Strata; that post has a score of 48 points and 51 comments.

The performance figures are claims presented in the repository description and the Reddit post; because they compare different baselines, they cannot be read as a single measurement result. The README is summarized as disclosing the measurement conditions and some hardware constraints. The license field is NOASSERTION, so the applicable license cannot be confirmed from the provided information alone. Therefore, performance and usage conditions should be judged within the publicly disclosed scope.