Devotd Github
NewInference & serving

ewhacc/tandem.cpp — Running Mixture-of-Experts Models on llama.cpp

ewhacc/tandem.cpp is a project that patches llama.cpp to run mixture-of-experts models that do not fit entirely in GPU memory.

Repositoryewhacc/tandem.cpp ↗

ewhacc/tandem.cpp is a project that patches llama.cpp to run mixture-of-experts models that do not fit entirely in GPU memory.

stars
★ 3
Forks
0
Language
Python
License
NOASSERTION
Read original source ↗View on GitHub ↗

ewhacc/tandem.cpp is a project that patches llama.cpp to support inference of mixture-of-experts models that do not fit entirely in GPU memory. There is no separate release information. According to the README, it caches frequently used experts on the GPU and computes the rest together with the CPU. The repository description states that it aims to improve model throughput.

The README presents throughput results for Qwen3.8-Flash-Next and other mixture-of-experts models together with the test environment. A table comparing several models on a single consumer computer and the installation and execution procedures are also available. It covers CUDA builds, configuration for environments with little spare memory, supported expert formats, and speculative decoding examples as well. It is therefore a repository that provides a concrete execution path for people who want to use llama.cpp in situations where the full model does not fit in GPU memory. The primary language is listed as Python.

However, the results have to be read together with the test environment shown in the README, and FACTS contains no separate release information. It has 3 stars, and the license is listed as NOASSERTION. Since the license terms cannot be confirmed from public information alone, the throughput results cannot currently be treated as a general performance guarantee, nor the project as a tool with clear usage conditions.