TokenSpeed

vLLM

AI GitHub Repos

#22

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

Petals

AI GitHub Repos

#105

Open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…Petals is an open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…

Official inference framework for 1-bit large language models (LLMs) developed by Microsoft. It enables efficient processing and deployment of LLMs, optimizing…BitNet is an official inference framework for 1-bit large language models (LLMs) developed by Microsoft. It enables efficient processing and deployment of LLMs, optimizing…

MNN

AI GitHub Repos

High-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

AirLLM

AI GitHub Repos

#134

GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLX

AI GitHub Repos

Array framework designed specifically for Apple silicon, enabling efficient machine learning operations. It allows developers to contribute to its development on GitHub,…MLX is an array framework designed specifically for Apple silicon, enabling efficient machine learning operations. It allows developers to contribute to its development on GitHub,…

FlashInfer

AI GitHub Repos

Kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…FlashInfer is a kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…

WebLLM

AI GitHub Repos

web-llm is a high-performance in-browser LLM inference engine designed to facilitate machine learning tasks directly within web environments. It allows developers to contribute…