srikanthbalaaji/dimmer-kv

srikanthbalaaji/dimmer-kv 是一个通过降低逐 token 精度来压缩 KV 缓存、并直接在压缩后的缓存上计算注意力的项目。

GitHub 项目说明: KV cache compression for LLM inference: 37× smaller KV cache at full-cache accuracy. Per-token precision on transform coding, question-time refinement, and attention over the compressed cache. PyTorch + Hugging Face.

星标

★ 2

复刻

0

语言

Python

许可证

MIT