Skip to main content
ToolPotion
AI Advisor
Submit a Tool (Free)

TensorFold

TensorFold is an open-source LLM inference engine designed for high-performance computing across various hardware platforms. It supports Metal, CUDA, and Vulkan, providing a flexible solution for deploying large language models. The project emphasizes efficient execution and offers tools for development and deployment, making it suitable for researchers and developers working with advanced AI models.

View Repository
Share
View Repository
Share

TensorFold is an open-source project hosted on GitHub, serving as an LLM (Large Language Model) inference engine. It is engineered to provide high-performance execution of large language models across diverse hardware architectures, specifically supporting Metal, CUDA, and Vulkan. This broad compatibility allows developers to deploy and run LLMs efficiently on a range of devices, from Apple's Metal-enabled hardware to NVIDIA's CUDA-powered GPUs and other devices supporting the Vulkan API.

Read More

The project is actively developed, with recent commits indicating continuous improvements and feature additions. It includes components for various aspects of LLM deployment, such as checkpoint and image preparation, orchestrator APIs for remote control, and tools for evaluation and scoring on different GPUs. TensorFold aims to offer a robust and flexible foundation for integrating LLM capabilities into applications and systems.

TensorFold's architecture is designed to facilitate the development and packaging of LLM runtimes. It provides documentation for optimizing performance and guides for evaluation. The repository structure includes directories for application code, assets, documentation, packaging tools, and web components, suggesting a comprehensive ecosystem for building and deploying LLM-based solutions. The project also features a Zig engine preview, indicating a focus on native performance and server-side capabilities for models like Nemotron 3.5 LLM.

With its focus on open-source development and support for multiple inference backends, TensorFold positions itself as a valuable resource for the AI community. It enables researchers and engineers to experiment with and deploy large language models without being constrained by specific hardware vendors, promoting broader accessibility and innovation in the field of AI inference.

  • CategoryMLOps & Model Deployment
  • Pricing ModelOpen source
  • DeploymentCloud, Self-hosted
  • PlatformsWeb

  • LLM Inference Engine

  • Supports Metal for Apple hardware

  • Supports CUDA for NVIDIA GPUs

  • Supports Vulkan for broad GPU compatibility

  • Open-source GitHub repository

  • Tools for checkpoint and image preparation

  • Orchestrator remote-control API

  • Automatic desktop runtime publishing for stable releases

  • GPU-agnostic text references and scoring tools

  • Native Metal engine and server preview for Nemotron 3.5 LLM

  • Comprehensive documentation for optimization and evaluation

  • Apache-2.0 License

  • Python engine integration (TensorFold 1.0.0)

  • Contributor agent pull request skills

  1. Clone Repository: Obtain the project by cloning the TensorFold GitHub repository to your local machine.

  2. Install Dependencies: Set up the necessary build environment and install any required dependencies as outlined in the project's documentation.

  3. Build Engine: Compile the TensorFold engine for your target hardware (Metal, CUDA, or Vulkan) using the provided build scripts.

  4. Configure Models: Prepare your LLM checkpoints and images according to the project's guidelines for inference.

  5. Run Inference: Execute LLM inference tasks using the TensorFold engine, leveraging its optimized performance for your chosen backend.

  6. Develop Applications: Integrate TensorFold into your applications using its APIs and tools to build custom LLM-powered solutions.

  7. Contribute: Engage with the open-source community by contributing to the project's development, documentation, or testing.

  • High-Performance LLM Inference
  • Cross-Platform AI Deployment
  • AI Model Development
  • Research and Experimentation
  • Custom LLM Application Building
  • GPU-Accelerated Text Processing

TensorFold Reviews

Loading...

Popular AI Tools Like TensorFold

TensorRT-LLM

AI GitHub Repos

TensorRT-LLM provides a Python API for defining Large Language Models, optimizing inference on NVIDIA GPUs. It includes components for creating Python and C++ runtimes to…

Machine Learning Platforms

Ranked #75 in AI GitHub Repos

View Tool

AI GitHub Repos

Provides a Python API for defining Large Language Models, optimizing inference on NVIDIA GPUs. It includes components for creating Python and C++ runtimes to…TensorRT-LLM provides a Python API for defining Large Language Models, optimizing inference on NVIDIA GPUs. It includes components for creating Python and C++ runtimes to…

Machine Learning Platforms
Ranked #75 in AI GitHub Repos

Soup

AI GitHub Repos

Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

MLOps & Model Deployment
View Tool

AI GitHub Repos

GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

AirLLM

AI GitHub Repos

AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLOps & Model Deployment

Ranked #72 in Machine Learning & Data Science

View Tool

AI GitHub Repos

GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLOps & Model Deployment
Ranked #72 in Machine Learning & Data Science

trnrt

AI GitHub Repos

trnrt is an open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…

MLOps & Model Deployment

Ranked #160 in MLOps & Model Deployment

View Tool

AI GitHub Repos

Open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…trnrt is an open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…

MLOps & Model Deployment
Ranked #160 in MLOps & Model Deployment

vLLM

AI GitHub Repos

vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

MLOps & Model Deployment

Ranked #22 in AI GitHub Repos

View Tool

AI GitHub Repos

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

FeaturedMLOps & Model Deployment
Ranked #22 in AI GitHub Repos

mlc-llm

AI GitHub Repos

mlc-llm is a universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…

MLOps & Model Deployment
View Tool

AI GitHub Repos

Universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…mlc-llm is a universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

TensorFlow

AI GitHub Repos

TensorFlow is an open-source machine learning framework designed for everyone. It provides a comprehensive ecosystem for building and deploying machine learning models, making it…

Machine Learning Platforms
View Tool

AI GitHub Repos

Open-source machine learning framework designed for everyone. It provides a comprehensive ecosystem for building and deploying machine learning models, making it…TensorFlow is an open-source machine learning framework designed for everyone. It provides a comprehensive ecosystem for building and deploying machine learning models, making it…

FeaturedMachine Learning Platforms
See the Top 250 AI GitHub Repos

llama-modchip

AI GitHub Repos

llama-modchip is a GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…

MLOps & Model Deployment
View Tool

AI GitHub Repos

GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…llama-modchip is a GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos
1 of 8