BigDL LLM

Description

The BigDL Project, specifically its BigDL-LLM component (now transitioning to IPEX-LLM), is a powerful library designed to optimize the performance of large language models (LLMs) across a range of Intel hardware, including CPUs and GPUs. It enables users to run LLMs with significantly reduced latency by leveraging advanced quantization techniques such as INT4, FP4, INT8, and FP8. This optimization is crucial for deploying complex AI models efficiently on diverse hardware configurations.

Built upon the foundational work of libraries like llama.cpp, gptq, bitsandbytes, and qlora, BigDL-LLM provides a robust framework for both inference and finetuning of PyTorch-based LLMs. Recent updates highlight expanded capabilities, including direct model loading from ModelScope, initial INT2 support for running large models on GPUs with limited VRAM, and integration with Text-Generation-WebUI for a graphical user experience. The library also introduces Self-Speculative Decoding, which offers a practical speedup for inference latency on Intel GPUs and CPUs.

For developers and researchers focused on model customization, BigDL-LLM offers comprehensive support for various finetuning methods on Intel GPUs, including LoRA, QLoRA, DPO, QA-LoRA, and ReLoRA. This allows for efficient adaptation of pre-trained models to specific tasks and datasets. The project has demonstrated impressive finetuning speeds, such as training LLaMA2-7B in under 30 minutes on multiple Intel GPUs. Furthermore, it supports direct loading of popular model formats like GGUF, AWQ, and GPTQ, and integrates with vLLM for continuous batching and FastChat for serving.

The BigDL ecosystem extends beyond LLMs, encompassing libraries for distributed data analytics and AI (Orca), transparent acceleration of TensorFlow and PyTorch (Nano), deep learning on Spark (DLlib), time series analysis (Chronos), recommendation systems (Friesian), and secure AI (PPML). BigDL-LLM is a key component for those looking to leverage the power of LLMs on Intel hardware, offering a flexible and performant solution for a wide array of AI applications.

BigDL LLM's Core Features

• Optimized LLM inference on Intel XPU (CPU, GPU)

• Supports INT4, FP4, INT8, FP8 quantization

• Low latency inference for PyTorch models

• Built on llama.cpp, gptq, bitsandbytes, qlora

• Direct loading of GGUF, AWQ, GPTQ models

• Comprehensive LLM finetuning on Intel GPU (LoRA, QLoRA, DPO, QA-LoRA, ReLoRA)

• Self-Speculative Decoding for ~30% speedup

• Supports vLLM continuous batching

• FastChat serving on Intel CPU and GPU

• Direct loading from ModelScope

• Initial INT2 support for large LLMs on 16GB VRAM GPUs

• Integration with Text-Generation-WebUI

Getting Started with BigDL LLM

1. 1

   Install BigDL-LLM via pip: `pip install --pre --upgrade bigdl-llm[all]` for CPU or `bigdl-llm[xpu]` for GPU.

2. 2

   Load Hugging Face Transformers models with INT4 optimizations using `AutoModelForCausalLM.from_pretrained`.

3. 3

   Run optimized models on Intel CPU by specifying the model path.

4. 4

   Run optimized models on Intel GPU by moving the model and input tensors to the 'xpu' device.

5. 5

   Configure finetuning parameters for LoRA, QLoRA, DPO, QA-LoRA, or ReLoRA.

6. 6

   Utilize FastChat or vLLM for serving optimized LLMs.

7. 7

   Integrate with LangChain for LLM application development.

BigDL LLM's Use Cases

• Low-latency LLM Inference
• LLM Finetuning
• Efficient Model Deployment
• AI Application Development
• Large Model Training on Limited VRAM
• Accelerated Text Generation

FAQ from BigDL LLM

• What is BigDL-LLM?

  BigDL-LLM is a library for running large language models (LLMs) on Intel XPU hardware, offering optimized inference and finetuning capabilities with low latency.

• What hardware does BigDL-LLM support?

  BigDL-LLM supports Intel XPU hardware, including laptops, CPUs, and GPUs (Arc, Flex, MAX).

• What quantization methods does BigDL-LLM support?

  BigDL-LLM supports INT4, FP4, INT8, and FP8 quantization for optimized inference.

• Can I finetune LLMs with BigDL-LLM?

  Yes, BigDL-LLM supports comprehensive LLM finetuning on Intel GPUs, including LoRA, QLoRA, DPO, QA-LoRA, and ReLoRA.

• What is the benefit of Self-Speculative Decoding in BigDL-LLM?

  Self-Speculative Decoding can bring approximately 30% speedup for FP16 and BF16 inference latency on Intel GPUs and CPUs, respectively.

• Does BigDL-LLM support loading models from other formats?

  Yes, BigDL-LLM supports directly loading GGUF, AWQ, and GPTQ models, as well as models from ModelScope.

• Is BigDL-LLM compatible with other LLM frameworks?

  Yes, BigDL-LLM has added support for LangChain, enabling its use in broader LLM application development.

• What are the performance claims for BigDL-LLM?

  BigDL-LLM aims for very low latency inference and has demonstrated fast finetuning times, such as LLaMA2-7B in 21 minutes on 8 Intel Max 1550 GPUs.

• Where can I find the latest development for BigDL-LLM?

  Future development for BigDL-LLM will move to the IPEX-LLM project.

• What are the installation requirements for BigDL-LLM?

  BigDL-LLM has been tested on Python 3.9, 3.10, and 3.11. Installation involves pip commands with specific extras for CPU or GPU.