New
srkrz23/vramkit — two-track VRAM compression for weights and the KV cache
srkrz23/vramkit is a Python repository implementing a lossless weight codec and KV cache compression that applies different quantization axes to keys and values.
Projects
srkrz23/vramkit
VRAM compression for LLM inference, measured on a MacBook: lossless weight codec (-33% vs 4-bit on real Qwen weights, beats zstd) + drop-in PyTorch KV-cache compression. Key finding: quantize KV keys per-CHANNEL (not per-token) -> int8 KV near-lossless, +0.1% real perplexity at 2x. ROCm/CUDA/MPS/CPU.
View on GitHub ↗- stars
- ★ 2
- Forks
- 0
- Language
- Python
- License
- —
- Latest release
- —
Posts
1No matching stories.