srikanthbalaaji/dimmer-kv is a Python project that compresses the key-value cache (KV cache) used in large language model inference at per-token precision. Instead of dropping tokens, it lowers the storage precision of each token, and when a new question arrives it enriches the details of the tokens that are needed. It also presents an implementation that computes attention directly from the compressed cache. The repository description says the cache can be made 37 times smaller while keeping the same accuracy as the original cache.
The core is a way of tuning the compression so that the information a question needs is preserved. The README describes using hot and cold tiers and progressive bit planes, and handling per-token precision with transform coding. It uses PyTorch and Hugging Face and applies an MIT license. The experiment reported in the README is under the conditions of Llama 3.2 1B and 8 thousand tokens. There, compressed to 3% the size of the FP16 cache, it recalled 21 of 24 hidden facts, and the repository reports this as the same result as the original cache.
However, the experiment results are self-reported by the repository, and the figures given are limited to a specific model and a specific set of conditions. There is nothing in the FACTS to support the view that the same compression ratio and recall results would come out on other models or tasks. So the 37x compression should be read as a result the project puts forward, and its practical applicability needs to be distinguished from validation outside those conditions.