Petals

Official inference framework for 1-bit large language models (LLMs) developed by Microsoft. It enables efficient processing and deployment of LLMs, optimizing…BitNet is an official inference framework for 1-bit large language models (LLMs) developed by Microsoft. It enables efficient processing and deployment of LLMs, optimizing…

PowerInfer

AI GitHub Repos

#208

High-speed large language model serving tool designed for local deployment. It enables efficient and rapid inference of large language models, making it suitable…PowerInfer is a high-speed large language model serving tool designed for local deployment. It enables efficient and rapid inference of large language models, making it suitable…

vLLM

AI GitHub Repos

#22

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

bitsandbytes

AI GitHub Repos

GitHub project focused on making large language models more accessible through k-bit quantization for PyTorch. It aims to optimize model performance and reduce…Bitsandbytes is a GitHub project focused on making large language models more accessible through k-bit quantization for PyTorch. It aims to optimize model performance and reduce…

ColossalAI

AI GitHub Repos

#65

Project aimed at making large AI models more affordable, faster, and accessible. It provides tools and frameworks to optimize AI model training and deployment,…ColossalAI is a project aimed at making large AI models more affordable, faster, and accessible. It provides tools and frameworks to optimize AI model training and deployment,…

FlashInfer

AI GitHub Repos

Kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…FlashInfer is a kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…

MNN

AI GitHub Repos

High-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

LocalAI

AI GitHub Repos

#44

Open-source AI engine that allows users to run various models, including LLMs, vision, voice, image, and video, on any hardware without requiring a GPU. This…LocalAI is an open-source AI engine that allows users to run various models, including LLMs, vision, voice, image, and video, on any hardware without requiring a GPU. This…