The latest release published by the intel/neural-compressor repository on GitHub is v3.10. It is an open-source model compression library written in Python, and it aims to reduce memory usage and computation cost by approximating the weights and operations of trained models to fewer bits (INT8, FP8, MXFP8, INT4, MXFP4, NVFP4).
The breadth of its coverage is this library's selling point. The README states that it supports PyTorch, TensorFlow and JAX, and the repository description lists quantization targeting PyTorch, TensorFlow and ONNX Runtime together with sparsity (setting some weights to zero). PyTorch, TensorFlow and JAX are widely used deep learning frameworks, and ONNX Runtime is a runtime that executes inference. It is structured to handle models from several frameworks in a single library; the license is Apache-2.0 and the repository has 2722 stars. The repository was created on July 21, 2020.
The techniques also branch into several lines. It aims to provide Static Quantization, Dynamic Quantization, SmoothQuant, Weight-Only Quantization, Quantization-Aware Training(QAT) and Mixed Precision, and through AutoRound integration it also supports advanced quantization for LLMs and VLMs (vision-language models). Weight-Only, which lowers only the weights to a lower precision, and QAT, which trains with quantization taken into account at the training stage, differ in where quantization is applied. Mixed Precision is the approach of lowering the precision of only some operations and leaving the rest as they are.
However, since the material contains no release-note entries for v3.10, it is impossible to pin down what changed in this release. The repository description introduces techniques that come ahead of low-bit LLM quantization, but it lists no performance comparison figures and no values for how much accuracy drops after quantization, so it is too early to rank it against other tools. Its actual usefulness can only be known by checking accuracy and processing speed before and after quantization on the target model and hardware, and that is as far as the material confirms.