kvmem/kvmem is a C++ project that keeps the KV cache holding the conversation during inference in host memory and brings it back to the GPU when needed, so it can handle long contexts. The latest version published on the GitHub repository is v0.18.0. The provided material contains no changelog for this version, so it cannot be confirmed what specifically is new.
The project uses the same shared memory, lookup and storage scheme for llama.cpp and NInfer, while keeping the inference of each backend separate. The default recipe is given as context 204800, GPU history budget 36864, generation reservation 16384, host KV 12288 MiB, and sink 5120 with recent 64 tokens. recent 64 is the minimum NInfer allows, and it differs in meaning from llama.cpp's default block of 128. The repository advises against copying those two values over as-is.
Verification is limited. The performance figures were confirmed on one RTX 5060 Ti 16 GB configuration, and NInfer's Windows release package has RC1 builds for hardware testing on sm75 and sm70, but physical GPU suitability verification is not yet complete. The repository holds the arXiv paper 2609.04852, C++ tests in 27 files, a backend version manifest and a SHA256 checksum file. This material alone still makes it hard to judge performance or compatibility on other hardware.