KV Cache
KV-cache implementations, virtualization, multiplexing, compression, and persistence for faster or larger-context inference; it excludes general-purpose disk caches and prompt-token billing tools.
| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| kvmem/kvmem | KVMem virtualizes KV memory across llama.cpp and NInfer, offloading cache state between host memory and GPU. | ★ 1.1K | C++ | — | v0.18.0 |