MNN

vLLM

AI GitHub Repos

#22

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

Official inference framework for 1-bit large language models (LLMs) developed by Microsoft. It enables efficient processing and deployment of LLMs, optimizing…BitNet is an official inference framework for 1-bit large language models (LLMs) developed by Microsoft. It enables efficient processing and deployment of LLMs, optimizing…

Baseten

AI Platforms

#72

Inference Platform allows users to deploy and scale open-source and custom AI models efficiently. It offers high-performance inference with dedicated infrastructure,…Baseten's Inference Platform allows users to deploy and scale open-source and custom AI models efficiently. It offers high-performance inference with dedicated infrastructure,…

AirLLM

AI GitHub Repos

#134

GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

mlc-llm

AI GitHub Repos

Universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…mlc-llm is a universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…

Soup

AI GitHub Repos

GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

omlx

AI GitHub Repos

#129

LLM inference server designed for Apple Silicon, featuring continuous batching and SSD caching. It is managed directly from the macOS menu bar, providing an efficient…omlx is an LLM inference server designed for Apple Silicon, featuring continuous batching and SSD caching. It is managed directly from the macOS menu bar, providing an efficient…

An AI infrastructure platform for developers to deploy, fine-tune, and run 200+ optimized LLMs and multimodal models through one OpenAI-compatible API with pay-as-you-go pricing.