Devotd Github
NewInference & serving

waybarrios/vllm-mlx — MLX Inference Server for Apple Silicon

Looks at the features of the waybarrios/vllm-mlx inference server for Apple Silicon and at the limits of the performance figures given in its documentation.

Repositorywaybarrios/vllm-mlx ↗

Looks at the features of the waybarrios/vllm-mlx inference server for Apple Silicon and at the limits of the performance figures given in its documentation.

GitHub description: High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

stars
★ 1.6K
Forks
233
Language
Python
License
Apache-2.0
Latest release
v0.6.0
Read original source ↗View on GitHub ↗

waybarrios/vllm-mlx is an inference server that serves language models with MLX on Apple Silicon. It supports OpenAI- and Anthropic-compatible interfaces, and the latest public release is v0.6.0. However, the provided record contains no changed items for this version, so the specific changes cannot be confirmed. Judging from the repository description and README, the project is a tool that connects several clients and features to a local inference environment.

The README explains several compatible interfaces, including continuous batching and a paged key-value cache. Continuous batching handles in-flight requests together instead of processing requests one at a time in sequence. The documentation also lists image, video and audio input, embeddings, MCP tool calls, and Claude Code support. It covers observability and benchmarking features, along with guidance on installation, checking the first response, and verifying clients. According to the repository description, the implementation language is Python and the license is Apache-2.0.

The performance table holds figures obtained on a specific device and configuration, and the README notes that results can vary with the execution environment. This material alone therefore cannot establish processing speed on other Apple Silicon devices or any performance advantage. The changelog for v0.6.0 is not among the provided facts either, so it is more accurate to treat this as material for checking the feature scope recorded in the README than as evidence of the specific changes in this release.