Devotd Github
NewInference & serving

wici-ai/inferssd — an MoE inference engine built on SSD streaming

Introduces the SSD streaming approach of InferSSD, which is built on llama.cpp, along with the models and access methods listed in the README.

Repositorywici-ai/inferssd ↗

Introduces the SSD streaming approach of InferSSD, which is built on llama.cpp, along with the models and access methods listed in the README.

GitHub description: Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

stars
★ 160
Forks
27
Language
C++
License
Apache-2.0
Latest release
v0.1.1
Read original source ↗View on GitHub ↗

InferSSD (wici-ai/inferssd) is a C++ engine built on llama.cpp that streams MoE experts from an SSD, letting it run models larger than RAM or VRAM. The latest release is v0.1.1. The provided information includes no change list explaining what differs in this version, so it cannot be said that any specific feature was newly added. The core of this repository is the way an SSD is put to use as part of the inference path when it is difficult to hold an entire model in RAM or VRAM. The project description presents this as an approach for running large models.

The README targets large mixture-of-experts models in GGUF format and currently lists DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open as supported targets. It also describes a browser interface for chatting with the model and an OpenAI-compatible API. Usage therefore comes down to two paths: talking to the model directly in the browser, or connecting through the compatible API. The installation instructions target macOS and Linux with NVIDIA GPUs. The license is Apache-2.0, and the primary language of the repository is C++. What can be confirmed within this scope is a specific model format, two ways to connect, and installation guidance for two operating environments.

However, the provided record contains no performance comparison figures. So it is not possible to judge from this information alone how far SSD streaming actually widens the memory limit, or what kind of speed it reaches on the supported models. The recorded momentum score is 5.0, while the technology, quality, and novelty scores were not provided. Rather than asserting the performance or the degree of improvement of v0.1.1, it is closer to the stage of examining applicability based on the supported models and the installation environment.