neroued/ninfer is a C++·CUDA-based local inference engine targeting some model checkpoints and GPUs. According to the repository description, it supports text and image/video generation, and provides a command-line tool and a server. There is no latest release registered. The repository is licensed under Apache-2.0 and has 3,228 stars.
Beyond running models, the engine also includes a server that applications can call. The server provides OpenAI- and Anthropic-compatible HTTP APIs. The README summary states support for the RTX 5090 and model-parallel execution using two DGX Spark units. It also includes CUDA graph decoding, speculative decoding, and a cache that reuses the prefix shared by multiple requests. One engine handles one model and processes up to eight concurrent requests. Tool calling, fixed-format output, and offline scoring are supported as well. Its own artifacts contain weights and configuration, tokenizers, and more.
However, the supported targets are limited to some model checkpoints and GPUs. From the information provided alone, it is not possible to confirm exactly which checkpoint and GPU combinations are supported. So while the feature list is broad, actual usability must be judged after checking whether the required model and hardware fall within the supported range.