b4rtaz/distributed-llama is a C++ engine that bundles personal devices such as home PCs and Raspberry Pi into a single cluster and runs one LLM split across it for inference. The latest release published on GitHub is v0.16.5. Tensor parallelism is an approach in which a single model is split across several devices, each handling part of the computation, and this project applies that approach across devices connected by Ethernet.
It is written in C++ and licensed under MIT. The repository description states that inference becomes faster the more devices you connect. It supports Linux, macOS and Windows, and is optimized for ARM and x86_64 AVX2 CPUs. It takes advantage of high-speed synchronization over Ethernet, and because a single root node coordinates multiple worker nodes, it handles not only distributed inference but also CLI chat and running an API server. The repository was created on December 4, 2023 and has 3112 stars.
However, the provided record does not list what changed in v0.16.5. There are no figures showing how throughput varies with device count, or how sensitive tensor parallelism is to Ethernet communication latency, and no validation data beyond the description in the README. It is difficult to judge from this information alone how much of a benefit you actually get when tying together a few personal devices.