Skip to main content
ToolPotion
AI Advisor
Submit a Tool (Free)

DeepSeek V3 JAX

AI GitHub ReposAI Models & LLMsOpen source · macOS, Linux

DeepSeek V3 JAX is an open-source implementation of the DeepSeek V3 model, including MLA, DeepSeekMoE, aux-loss-free balancing, and MTP. Developed in 190 lines of pure JAX, this project demonstrates how to train and sample the model, even on a MacBook Air. It provides a concise and efficient way to explore advanced large language model architectures.

View Repository
Share
View Repository
Share

DeepSeek V3 JAX is a compact and efficient open-source implementation of the DeepSeek V3 large language model architecture. This project, developed by Vaibhaav Tiwari, distills the complex DeepSeek V3 design, which incorporates features like Multi-Layer Attention (MLA), DeepSeekMoE (Mixture-of-Experts), aux-loss-free balancing, and Multi-Task Pre-training (MTP), into approximately 190 lines of pure JAX code.

Read More

The repository provides a comprehensive guide for understanding, training, and sampling the DeepSeek V3 model. It includes scripts for model definition, training loops, data preparation, and sampling, making it accessible for researchers and developers interested in experimenting with advanced LLM architectures. A notable aspect of this implementation is its demonstrated ability to be trained on a MacBook Air, highlighting its efficiency and potential for local development and experimentation.

The project emphasizes clarity and conciseness, offering a minimalist approach to implementing a sophisticated model. It serves as an educational resource for those looking to delve into the specifics of DeepSeek V3's components and their JAX implementation. The repository also includes tests to ensure correctness and provides a full reproduction guide, complete with verified numbers and results, allowing users to replicate the reported performance.

Beyond the core model implementation, DeepSeek V3 JAX also addresses practical aspects such as ROCm (AMD GPU) setup for training on Linux, indicating its adaptability to different hardware environments. The inclusion of a detailed write-up on vaibhaav.in/writing/deepseek-v3-jax offers further insights into the design choices and theoretical underpinnings of the project, making it a valuable resource for both practical application and academic study.

  • CategoryAI Models & LLMs
  • Pricing ModelOpen source
  • DeploymentSelf-hosted
  • PlatformsmacOS, Linux

  • DeepSeek V3 model implementation in JAX

  • Includes Multi-Layer Attention (MLA)

  • Incorporates DeepSeekMoE architecture

  • Features aux-loss-free balancing

  • Supports Multi-Task Pre-training (MTP)

  • Concise codebase (approx. 190 lines of JAX)

  • Demonstrated training on MacBook Air

  • Scripts for model, training, sampling, and data preparation

  • Includes tests for verification

  • Full reproduction guide with verified results

  • ROCm (AMD GPU) setup for Linux training

  • MIT License for open-source use

  1. Clone the repository: Obtain the project files from GitHub.

  2. Install dependencies: Use `requirements.txt` to set up the necessary Python packages.

  3. Prepare data: Run `prepare_data.py` to get your dataset ready for training.

  4. Train the model: Execute `train.py` to start the model training process.

  5. Sample outputs: Use `sample.py` to generate text from the trained model.

  6. Run tests: Verify the implementation's correctness with `tests.py`.

  • LLM Architecture Research
  • Efficient Model Implementation
  • Local LLM Training
  • JAX Development Practice
  • Open-Source Contribution
  • Educational Resource

DeepSeek V3 JAX Reviews

Loading...

Popular AI Tools Like DeepSeek V3 JAX

DeepSeek-V3

AI GitHub Repos

DeepSeek-V3 is an open-weight Mixture-of-Experts language model from DeepSeek, with 671 billion total parameters and 37 billion activated per token. Pre-trained on 14.8 trillion…

AI Models & LLMs
View Tool

AI GitHub Repos

Open-weight Mixture-of-Experts language model from DeepSeek, with 671 billion total parameters and 37 billion activated per token. Pre-trained on 14.8 trillion…DeepSeek-V3 is an open-weight Mixture-of-Experts language model from DeepSeek, with 671 billion total parameters and 37 billion activated per token. Pre-trained on 14.8 trillion…

AI Models & LLMs
See the Top 250 AI GitHub Repos

DeepSeek-V3 is a Mixture-of-Experts language model with 671 billion parameters, designed for efficient inference and cost-effective training. It excels in various benchmarks,…

AI Models & LLMs

Ranked #42 in LLMs

View Tool

Mixture-of-Experts language model with 671 billion parameters, designed for efficient inference and cost-effective training. It excels in various benchmarks,…DeepSeek-V3 is a Mixture-of-Experts language model with 671 billion parameters, designed for efficient inference and cost-effective training. It excels in various benchmarks,…

FeaturedAI Models & LLMs
Ranked #42 in LLMs

OpenLLaMA

AI GitHub Repos

OpenLLaMA is a permissively licensed, open-source reproduction of Meta AI's LLaMA 7B model. Trained on the RedPajama dataset, it offers 3B, 7B, and 13B parameter versions,…

AI Models & LLMs
View Tool

AI GitHub Repos

Permissively licensed, open-source reproduction of Meta AI's LLaMA 7B model. Trained on the RedPajama dataset, it offers 3B, 7B, and 13B parameter versions,…OpenLLaMA is a permissively licensed, open-source reproduction of Meta AI's LLaMA 7B model. Trained on the RedPajama dataset, it offers 3B, 7B, and 13B parameter versions,…

AI Models & LLMs
See the Top 250 AI GitHub Repos

DeepSeek is an AI research company dedicated to developing general artificial intelligence. It offers open-source frontier LLMs like DeepSeek-V4, DeepSeek-R1, and DeepSeek-Coder,…

AI Models & LLMs

Ranked #2 in LLMs

View Tool

AI research company dedicated to developing general artificial intelligence. It offers open-source frontier LLMs like DeepSeek-V4, DeepSeek-R1, and DeepSeek-Coder,…DeepSeek is an AI research company dedicated to developing general artificial intelligence. It offers open-source frontier LLMs like DeepSeek-V4, DeepSeek-R1, and DeepSeek-Coder,…

FeaturedAI Models & LLMs
Ranked #2 in LLMs

DeepSeek-R1

AI GitHub Repos

DeepSeek-R1 is a GitHub project focused on AI-driven solutions. It offers a variety of repositories that enhance AI capabilities, including plugins, sparse attention kernels, and…

AI Models & LLMs
View Tool

AI GitHub Repos

GitHub project focused on AI-driven solutions. It offers a variety of repositories that enhance AI capabilities, including plugins, sparse attention kernels, and…DeepSeek-R1 is a GitHub project focused on AI-driven solutions. It offers a variety of repositories that enhance AI capabilities, including plugins, sparse attention kernels, and…

AI Models & LLMs
See the Top 250 AI GitHub Repos

FNet

AI GitHub Repos

FNet is an efficient Transformer-like encoder architecture that replaces self-attention with Fourier Transforms. Developed by Google Research, it offers a high-performance…

AI Models & LLMs
View Tool

AI GitHub Repos

Efficient Transformer-like encoder architecture that replaces self-attention with Fourier Transforms. Developed by Google Research, it offers a high-performance…FNet is an efficient Transformer-like encoder architecture that replaces self-attention with Fourier Transforms. Developed by Google Research, it offers a high-performance…

AI Models & LLMs
See the Top 250 AI GitHub Repos

A free online demo of the open-source DeepSeek V3 model (671B parameters) that runs in your browser with no registration, plus links to run the model locally for personal and…

AI Models & LLMs
View Tool

A free online demo of the open-source DeepSeek V3 model (671B parameters) that runs in your browser with no registration, plus links to run the model locally for personal and…

AI Models & LLMs
See the Top 250 AI Apps

LlamaFactory

AI GitHub Repos

LlamaFactory is a GitHub repository focused on the unified and efficient fine-tuning of over 100 large language models (LLMs) and vision-language models (VLMs). It aims to…

Machine Learning Platforms

Ranked #13 in AI GitHub Repos

View Tool

AI GitHub Repos

GitHub repository focused on the unified and efficient fine-tuning of over 100 large language models (LLMs) and vision-language models (VLMs). It aims to…LlamaFactory is a GitHub repository focused on the unified and efficient fine-tuning of over 100 large language models (LLMs) and vision-language models (VLMs). It aims to…

FeaturedMachine Learning Platforms
Ranked #13 in AI GitHub Repos
1 of 8