Skip to main content
ToolPotion
AI Advisor
Submit a Tool (Free)

smolmaxxing

smolmaxxing is a GitHub project focused on optimizing the execution of various large language models (LLMs) on resource-constrained devices. It provides benchmarks, tools, and documentation to enable efficient LLM inference on small devices like phones, iPads, and TV boxes. The project aims to maximize performance and compatibility across a range of hardware, making advanced AI accessible on everyday gadgets.

View Repository
Share
View Repository
Share

smolmaxxing is an open-source GitHub repository dedicated to pushing the boundaries of running large language models (LLMs) on the smallest possible devices. The project's core mission is to maximize the ability to deploy and execute different LLMs efficiently on hardware typically considered underpowered for such tasks, including smartphones, tablets, and smart TV boxes.

Read More

The repository contains a comprehensive collection of resources, including benchmarks, data, documentation, and tools. It showcases performance metrics for various LLMs, such as Qwen3-Next-80B-A3B and Qwen3.6-35B-A3B, running on specific devices like a OnePlus phone or a Xiaomi TV Box S. The project meticulously tracks token per second (tok/s) performance, demonstrating the feasibility of running models ranging from 260K to 9B parameters on these compact devices.

smolmaxxing provides a structured approach to experimentation, with dedicated directories for assets, benchmarks, data, and documentation. It includes patches and tools to facilitate the optimization process, along with screenshots illustrating the project's progress. The `README.md` file serves as a central hub, featuring a scoreboard of the best-performing models per size class and links to models hosted on Hugging Face.

This project is ideal for developers, researchers, and enthusiasts interested in edge AI, on-device machine learning, and optimizing LLM inference for low-power environments. It fosters collaboration, with contributions from multiple developers, and includes a `CONTRIBUTING.md` guide for those looking to submit their device benchmarks or code improvements. The `LICENSE` file indicates an MIT license, promoting open usage and modification.

  • CategoryMLOps & Model Deployment
  • Pricing ModelOpen source
  • DeploymentCloud, Self-hosted
  • PlatformsWeb

  • Optimized LLM inference on small devices

  • Benchmarks for various LLMs on diverse hardware

  • Support for models from 260K to 9B parameters

  • Performance tracking (tokens per second)

  • Documentation for device-specific optimizations

  • Tools and patches for LLM deployment

  • Scoreboard of best-performing models

  • Links to Hugging Face models

  • Community contributions and issue forms

  • MIT License for open-source use

  • Examples for phones, iPads, and TV boxes

  • Repository for ongoing research and development

  1. Clone Repository: Use `git clone` to get the smolmaxxing project onto your local machine.

  2. Explore Documentation: Review the `docs/` directory and `README.md` for project overview, device specifics, and setup instructions.

  3. Review Benchmarks: Examine the `bench/` and `data/` directories to understand existing LLM performance on various small devices.

  4. Install Dependencies: Identify and install any necessary libraries or tools required for running LLMs on your target device.

  5. Apply Patches/Tools: Utilize provided patches and tools from the `patches/` and `tools/` directories to optimize LLM execution.

  6. Run Experiments: Follow the guidelines in `EXPERIMENT.md` to conduct your own LLM inference tests on small devices.

  7. Contribute Findings: Use the `Submit your device` issue form or `CONTRIBUTING.md` to share your results or code improvements.

  • Edge AI Development
  • On-Device LLM Inference
  • Performance Benchmarking
  • Resource-Constrained AI
  • AI for IoT Devices
  • Mobile AI Application

smolmaxxing Reviews

Loading...

Popular AI Tools Like smolmaxxing

MNN

AI GitHub Repos

MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

MLOps & Model Deployment
View Tool

AI GitHub Repos

High-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…MNN is a high-performance, lightweight inference engine developed by Alibaba. It is designed to optimize on-device LLMs and Edge AI applications, providing fast and efficient…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

BigMoeOnEdge

AI GitHub Repos

BigMoeOnEdge enables running MoE models larger than your RAM on various devices, including phones and laptops. It uses an adaptive streaming optimizer to ensure lossless…

MLOps & Model Deployment
View Tool

AI GitHub Repos

Enables running MoE models larger than your RAM on various devices, including phones and laptops. It uses an adaptive streaming optimizer to ensure lossless…BigMoeOnEdge enables running MoE models larger than your RAM on various devices, including phones and laptops. It uses an adaptive streaming optimizer to ensure lossless…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

trnrt

AI GitHub Repos

trnrt is an open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…

MLOps & Model Deployment

Ranked #160 in MLOps & Model Deployment

View Tool

AI GitHub Repos

Open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…trnrt is an open-source LLM inference engine designed for AWS Trainium, offering NKI megakernels and supporting a 1M-token context. Developed by RunAnywhere, it provides an…

MLOps & Model Deployment
Ranked #160 in MLOps & Model Deployment

vLLM

AI GitHub Repos

vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

MLOps & Model Deployment

Ranked #23 in AI GitHub Repos

View Tool

AI GitHub Repos

High-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…vllm-project/vllm is a high-throughput and memory-efficient inference and serving engine designed for large language models (LLMs). It optimizes performance while minimizing…

FeaturedMLOps & Model Deployment
Ranked #23 in AI GitHub Repos

llama-modchip

AI GitHub Repos

llama-modchip is a GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…

MLOps & Model Deployment
View Tool

AI GitHub Repos

GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…llama-modchip is a GitHub project enabling users to run large language models like 284B DeepSeek-V4-Flash and 30B MoEs on a pure 64GB CPU without GPUs. It supports high token per…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

mlc-llm

AI GitHub Repos

mlc-llm is a universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…

MLOps & Model Deployment
View Tool

AI GitHub Repos

Universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…mlc-llm is a universal LLM deployment engine that utilizes machine learning compilation to streamline the deployment process. It is designed to enhance the efficiency and…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

Soup

AI GitHub Repos

Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

MLOps & Model Deployment
View Tool

AI GitHub Repos

GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…Soup is a GitHub repository that enables the fine-tuning of large language models (LLMs) using a single YAML file. It utilizes layer streaming to train an 8 billion parameter…

MLOps & Model Deployment
See the Top 250 AI GitHub Repos

AirLLM

AI GitHub Repos

AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLOps & Model Deployment

Ranked #72 in Machine Learning & Data Science

View Tool

AI GitHub Repos

GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…AirLLM is a GitHub project that enables 70 billion parameter inference using a single 4GB GPU. It is designed to optimize large language model performance on limited hardware,…

MLOps & Model Deployment
Ranked #72 in Machine Learning & Data Science
1 of 8