Description
Intel® Neural Compressor is a powerful open-source Python library designed to enhance the performance of deep learning models through various compression techniques. It seamlessly integrates with mainstream frameworks like PyTorch, TensorFlow, and JAX, making it a versatile tool for developers and researchers.
Read More
The library provides a comprehensive suite of model compression methods, including Static Quantization, Dynamic Quantization, SmoothQuant, Weight-Only Quantization, Quantization-Aware Training, and Mixed Precision. A key capability is its support for advanced quantization of Large Language Models (LLMs) and Vision-Language Models (VLMs) such as LLaMA, Qwen, and DeepSeek, leveraging integration with AutoRound for optimized low-precision data types.
Intel® Neural Compressor is engineered for broad hardware compatibility. It offers extensive testing and support for a wide range of Intel hardware, including Intel Gaudi AI Accelerators, Intel Core Ultra Processors, Intel Xeon Scalable Processors, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series. Additionally, it provides limited testing support for AMD CPUs, ARM CPUs, and NVIDIA GPUs.
The library's documentation details installation procedures for different frameworks and hardware. Users can install the Neural Compressor with specific framework dependencies via pip, such as `neural-compressor-pt` for PyTorch or `neural-compressor-tf` for TensorFlow. For JAX, a build-from-source installation method is available. The documentation also includes getting started guides with example code for techniques like FP8 Quantization on Intel Gaudi2 AI Accelerators and weight-only LLM loading.
Recent updates highlight experimental support for FP8 quantization in Keras/JAX, FP8 KV cache/Attention static quantization with AutoRound, and NVFP4/MXFP4 quantization. The project actively encourages contributions and collaborations, with clear guidelines for bug reports, feature requests, and research ideas via GitHub Issues and email.
- CategoryMachine Learning Platforms
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
Intel® Neural Compressor's Core Features
Supports PyTorch, TensorFlow, and JAX frameworks
Offers Static Quantization, Dynamic Quantization, SmoothQuant, Weight-Only Quantization
Enables Quantization-Aware Training and Mixed Precision
Advanced quantization for LLMs and VLMs (e.g., LLaMA, Qwen)
Integration with AutoRound for optimized low-precision data types
Extensive support for Intel hardware (Gaudi, Xeon, Core Ultra, Data Center GPUs)
Limited support for AMD CPU, ARM CPU, and NVIDIA GPU
Experimental FP8 quantization support for Keras/JAX
Experimental FP8 KV cache/Attention static quantization
Experimental NVFP4 and MXFP4 quantization support
Weight-only large language model loading
Getting Started with Intel® Neural Compressor
Install framework dependencies: Choose and install necessary packages based on your deployment environment (e.g., `pip install neural-compressor-pt`).
Configure quantization: Define quantization configurations, such as FP8Config, for your model.
Prepare model: Use the `prepare` function to integrate quantization configurations into your model.
Convert model: Apply the `convert` function to finalize the model compression.
Deploy and optimize: Integrate the compressed model into your application for improved inference performance.
Intel® Neural Compressor's Use Cases
- LLM Quantization
- VLM Optimization
- Hardware Acceleration
- Framework Integration
- Performance Tuning
- Mixed Precision Training
