Skip to main content
ToolPotion
AI Advisor
Submit a Tool (Free)

inferssd

AI GitHub ReposMLOps & Model DeploymentOpen source · Windows, macOS

inferssd is an open-source GitHub project designed to run large AI models that exceed the capacity of your system's RAM or VRAM. It achieves this by efficiently streaming Mixture-of-Experts (MoE) components directly from SSD storage. Built upon the llama.cpp framework, inferssd enables local execution of advanced models on more constrained hardware, making powerful AI accessible without requiring extensive memory upgrades.

View Repository
Share
View Repository
Share

inferssd is a GitHub project focused on enabling the execution of large AI models, specifically those with Mixture-of-Experts (MoE) architectures, even when their size surpasses the available RAM or VRAM. The core innovation lies in its ability to stream MoE experts from SSD storage, effectively using the disk as an extension of system memory. This approach significantly lowers the hardware requirements for running advanced models locally, making them accessible to a broader range of users and systems.

Read More

The project is built on the foundation of llama.cpp, a high-performance inference engine for large language models. By leveraging llama.cpp, inferssd benefits from an optimized base for model execution, while extending its capabilities to handle models that would otherwise be too large for typical consumer-grade hardware. This is particularly valuable for researchers, developers, and enthusiasts who want to experiment with or deploy large models without investing in expensive, high-capacity GPUs or vast amounts of RAM.

Key capabilities include efficient memory management that prioritizes streaming model components as needed, rather than loading the entire model into volatile memory at once. This dynamic loading mechanism ensures that only the currently active 'experts' or model segments reside in RAM/VRAM, minimizing the memory footprint. The project also demonstrates ongoing development, with recent commits indicating support for Windows SSD inference on laptops and optimizations for various hardware configurations, including unified-memory GPUs on macOS.

inferssd aims to democratize access to large AI models by overcoming common hardware limitations. Its open-source nature encourages community contributions and further development, fostering innovation in local AI inference. The project's structure, with distinct 'engine' and 'framework' components, suggests a modular design that could facilitate future expansions and integrations with different model architectures or hardware platforms.

  • CategoryMLOps & Model Deployment
  • Pricing ModelOpen source
  • DeploymentSelf-hosted
  • PlatformsWindows, macOS

  • Run models larger than RAM or VRAM

  • Stream Mixture-of-Experts (MoE) from SSD

  • Built on llama.cpp for efficient inference

  • Supports large AI model execution on constrained hardware

  • Optimized memory management for MoE models

  • Enables local inference without extensive memory upgrades

  • Ongoing development with recent commits

  • Support for Windows SSD inference on laptops

  • Optimizations for unified-memory GPUs (e.g., macOS)

  • Modular 'engine' and 'framework' architecture

  • Apache-2.0 License for open-source use

  • Publicly available on GitHub for collaboration

  1. Clone Repository: Use `git clone` to download the inferssd project from GitHub to your local machine.

  2. Install Dependencies: Follow the project's documentation (e.g., in `docs` or `README.md`) to install necessary build tools and libraries, potentially including CMake and specific compilers.

  3. Configure Build: Utilize CMake to configure the build process, specifying your target platform and any desired options.

  4. Compile Project: Compile the source code using your chosen build system (e.g., Make or Visual Studio) to generate the inferssd executables.

  5. Prepare Models: Convert or obtain MoE models compatible with llama.cpp and inferssd's streaming mechanism, ensuring they are correctly formatted for SSD inference.

  6. Execute Inference: Run the compiled inferssd application with your prepared models, allowing it to stream experts from SSD for inference.

  7. Monitor Performance: Observe resource usage and inference speed, potentially adjusting configurations for optimal performance on your specific hardware.

  • Local LLM Inference
  • MoE Model Deployment
  • AI Research & Development
  • Edge AI Applications
  • Cost-Effective AI Prototyping

inferssd Reviews

Loading...

Popular AI Tools Like inferssd

BigMoeOnEdge

AI GitHub Repos

BigMoeOnEdge enables running MoE models larger than your RAM on various devices, including phones and laptops. It uses an adaptive streaming optimizer to ensure lossless…

MLOps & Model Deployment
View Tool

AI GitHub Repos

Enables running MoE models larger than your RAM on various devices, including phones and laptops. It uses an adaptive streaming optimizer to ensure lossless…BigMoeOnEdge enables running MoE models larger than your RAM on various devices, including phones and laptops. It uses an adaptive streaming optimizer to ensure lossless…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

llama-modchip

AI GitHub Repos

llama-modchip is a GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…

MLOps & Model Deployment
View Tool

AI GitHub Repos

GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…llama-modchip is a GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

vLLM

AI GitHub Repos

vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

MLOps & Model Deployment

Ranked #23 in AI GitHub Repos

View Tool

AI GitHub Repos

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

FeaturedMLOps & Model Deployment
Ranked #23 in AI GitHub Repos

trnrt

AI GitHub Repos

trnrt is an open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…

MLOps & Model Deployment

Ranked #160 in MLOps & Model Deployment

View Tool

AI GitHub Repos

Open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…trnrt is an open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…

MLOps & Model Deployment
Ranked #160 in MLOps & Model Deployment

Soup

AI GitHub Repos

Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

MLOps & Model Deployment
View Tool

AI GitHub Repos

GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

MNN

AI GitHub Repos

MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

MLOps & Model Deployment
View Tool

AI GitHub Repos

High-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

LMDeploy

AI GitHub Repos

LMDeploy is a toolkit designed for compressing, deploying, and serving large language models (LLMs). It provides developers with the necessary tools to efficiently manage LLMs,…

MLOps & Model Deployment

Ranked #57 in AI GitHub Repos

View Tool

AI GitHub Repos

Toolkit designed for compressing, deploying, and serving large language models (LLMs). It provides developers with the necessary tools to efficiently manage LLMs,…LMDeploy is a toolkit designed for compressing, deploying, and serving large language models (LLMs). It provides developers with the necessary tools to efficiently manage LLMs,…

MLOps & Model Deployment
Ranked #57 in AI GitHub Repos

AirLLM

AI GitHub Repos

AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLOps & Model Deployment

Ranked #72 in Machine Learning & Data Science

View Tool

AI GitHub Repos

GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLOps & Model Deployment
Ranked #72 in Machine Learning & Data Science
1 of 8