Devotd Github
NewInference & serving

dylanatl15/engine-lab — llama.cpp patches for a single Radeon VII

dylanatl15/engine-lab is a repository that documents how to run large mixture-of-experts models on a single 16GB Radeon VII, through llama.cpp patches, run tools, and measurement logs.

Repositorydylanatl15/engine-lab ↗

dylanatl15/engine-lab is a repository that documents how to run large mixture-of-experts models on a single 16GB Radeon VII, through llama.cpp patches, run tools, and measurement logs.

GitHub description: Run a 125B-class MoE (Qwen3.8-Flash-Next) at ~27 t/s on ONE 16 GB Radeon VII: expert-cache patches, benchmarks, profiling notes and dead ends

stars
★ 0
Forks
0
Language
C++
License
NOASSERTION
Read original source ↗View on GitHub ↗

dylanatl15/engine-lab is not a new inference engine but a collection of patches applied to the existing llama.cpp. There is no published release and no version information. The repository description states that it runs the mixture-of-experts (MoE) model Qwen3.8-Flash-Next at roughly 27 t/s on a single 16GB Radeon VII. The README lays out an approach that keeps model weights in host memory and caches frequently used experts in graphics memory, along with related experiments.

This repository locates the cause of the performance degradation in weight placement rather than in a lack of compute, and covers patches that improve caching and data transfer. It compares fixed-buffer transfers, parallel copies, and demand paging, and also weighs separate kernel fusion against changes to execution scheduling. It ships run tools and a test harness together with measurement logs and failure cases, showing the process of adapting llama.cpp to a specific environment. It is closer to material for experimenting with the execution paths of existing code than to a new engine.

The performance figures are results obtained on a specific graphics card and configuration. The repository description cites about 27 t/s, but the investigation logs record a value of up to 24 tokens per second, so the numbers do not match. It is therefore hard to generalize them into a performance baseline for other hardware. The license is listed as NOASSERTION and cannot be confirmed, and there is no public release either. For now, it is more reasonable to treat this as an experimental log for examining reproduction conditions than as a general-purpose distribution.