AirLLM

vLLM

AI GitHub Repos

#22

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

Soup

AI GitHub Repos

GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

PowerInfer

AI GitHub Repos

#208

High-speed large language model serving tool designed for local deployment. It enables efficient and rapid inference of large language models, making it suitable…PowerInfer is a high-speed large language model serving tool designed for local deployment. It enables efficient and rapid inference of large language models, making it suitable…

FlashInfer

AI GitHub Repos

Kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…FlashInfer is a kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…

ColossalAI

AI GitHub Repos

#65

Project aimed at making large AI models more affordable, faster, and accessible. It provides tools and frameworks to optimize AI model training and deployment,…ColossalAI is a project aimed at making large AI models more affordable, faster, and accessible. It provides tools and frameworks to optimize AI model training and deployment,…

MNN

AI GitHub Repos

High-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

mlc-llm

AI GitHub Repos

Universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…mlc-llm is a universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…

Petals

AI GitHub Repos

#105

Open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…Petals is an open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…