Inference Memory and Cache
Technologies that reduce or virtualize model weights, activations, attention state, or KV-cache memory during inference; it excludes lossy model quantization done before deployment and ordinary RAM management.
Subfields
KV Cache
KV-cache implementations, virtualization, multiplexing, compression, and persistence for faster or larger-context inference; it excludes general-purpose disk caches and prompt-token billing tools.
1 projectsModel Memory Compression
Runtime techniques that reduce the memory footprint of loaded model weights or state with minimal quality loss; it excludes offline quantization-format research and inference benchmarking suites.
2 projects| Project | What it does | Stars | Language | License | Latest release |
|---|---|---|---|---|---|
| kvmem/kvmem | KVMem virtualizes KV memory across llama.cpp and NInfer, offloading cache state between host memory and GPU. | ★ 1.1K | C++ | — | v0.18.0 |
| surya-koritala/glyd | A lossless runtime codec for BF16 weights and KV-cache data that reduces GPU memory and reconstructs exact values. | ★ 255 | Rust | NOASSERTION | v0.32.1 |
| srkrz23/vramkit | Provides experimental compression for LLM inference weights and KV caches to reduce VRAM use. | ★ 3 | Python | — | — |