Devotd Github
NewInference & serving

defai-digital/ax-engine v7.8.0 — Multi-model serving on Apple Silicon

defai-digital/ax-engine is a runtime environment that serves multiple language models on Apple Silicon, and its latest release is v7.8.0.

Repositorydefai-digital/ax-engine ↗

defai-digital/ax-engine is a runtime environment that serves multiple language models on Apple Silicon, and its latest release is v7.8.0.

GitHub description: One Mac process. Many models. Real speed. Multi-model LLM serving with prefix reuse, MTP acceleration, and OpenAI APIs — built for Apple Silicon, measured against mlx-lm and llama.cpp.

stars
★ 32
Forks
2
Language
Rust
License
Apache-2.0
Latest release
v7.8.0
Read original source ↗View on GitHub ↗

defai-digital/ax-engine is an MLX-based runtime environment that serves multiple language models on Apple Silicon. The latest release is v7.8.0. FACTS contains no specific record of what changed in this version, so the changes cannot be confirmed. The repository description presents multi-model serving, prefix reuse, and MTP acceleration as its main features, and states that it also provides an OpenAI API.

According to the README, it separates per-model packages and optional features, and covers residency of multiple models, request path selection, scheduling, and key-value caching. Prefix reuse is a way to reuse the computation of the shared prefix that requests have in common. Key-value caching is a feature that stores and reuses the key and value information the model produces while processing tokens. The repository description says it takes mlx-lm and llama.cpp as comparison targets. This project is written in Rust and is licensed under Apache-2.0.

However, the README's claims about performance and quality are limited to specific hardware and model combinations. Some acceleration features are described as still awaiting verification or as experimental optional features. The specific results of the comparative measurements are not in FACTS, and the change log for v7.8.0 was not given either. Therefore, within the verifiable scope, the feature composition and supported direction can be examined, but there is not enough basis to judge performance superiority or the improvements of this release.