LLM architecture from scratch

AI GitHub Repos AI Models & LLMsOpen source · Web

llm-architecture-from-scratch is a GitHub repository by Yeonwoo Sung that focuses on implementing large language model architectures from scratch using PyTorch. It provides a comprehensive collection of models and tools for developers interested in building and experimenting with LLMs.

View Repository

Description

llm-architecture-from-scratch is a public GitHub repository created by Yeonwoo Sung, dedicated to the implementation of large language model (LLM) architectures using PyTorch. This repository serves as a valuable resource for developers and researchers who are interested in constructing LLMs from the ground up. The repository includes a variety of models such as deepseekv3, gemma3, gpt2-xl, llama-3-2-3b, llama3-8b, llama4_maverick, mistral_3_1, olmo2, phi4, qwen3_235b_a22b, qwen3_32b, and xlstm. Each model is accompanied by detailed commit messages that provide insights into the development process, such as feature implementations, refactoring, and documentation updates.

The repository is actively maintained, with the latest commit made in October 2026, ensuring that the content remains up-to-date with the latest advancements in LLM architectures. Users can explore the repository to understand the intricacies of LLM design and gain hands-on experience by experimenting with the provided code.

This project is particularly beneficial for AI researchers, machine learning engineers, and developers who aim to deepen their understanding of LLMs and contribute to the field of natural language processing. By offering a diverse set of models and comprehensive documentation, llm-architecture-from-scratch empowers users to build, test, and optimize their own LLMs, fostering innovation and collaboration within the AI community.

CategoryAI Models & LLMs
Pricing ModelOpen source
DeploymentCloud, Self-hosted
PlatformsWeb

LLM architecture from scratch's Core Features

Implementation of various LLM architectures
Models include deepseekv3, gemma3, gpt2-xl, and more
Developed using PyTorch
Detailed commit messages for each model
Active maintenance with recent updates
Comprehensive documentation
Resource for AI researchers and developers
Facilitates experimentation with LLMs

Getting Started with LLM architecture from scratch

Clone: Download the repository from GitHub
Install dependencies: Set up the required Python packages
Configure: Adjust model parameters as needed
Execute: Run the models to test their functionality
Optimize: Experiment with different configurations to improve performance

LLM architecture from scratch's Use Cases

Model Development
Research
Education
Collaboration
Optimization

FAQ from LLM architecture from scratch

LLM architecture from scratch Reviews

Popular AI Tools Like LLM architecture from scratch

LLMs-from-scratch

AI GitHub Repos

LLMs-from-scratch is a GitHub project that guides users in implementing a ChatGPT-like language model using PyTorch. It provides a step-by-step approach to building large language…

AI Models & LLMs

View Tool

GitHub project that guides users in implementing a ChatGPT-like language model using PyTorch. It provides a step-by-step approach to building large language…LLMs-from-scratch is a GitHub project that guides users in implementing a ChatGPT-like language model using PyTorch. It provides a step-by-step approach to building large language…

AI Models & LLMs

See the Top 250 AI GitHub Repos

Happy LLM

AI GitHub Repos

Happy LLM is an open-source project aimed at building large language models from scratch. It provides resources and tools for developers to contribute and collaborate on model…

AI Models & LLMs

View Tool

Open-source project aimed at building large language models from scratch. It provides resources and tools for developers to contribute and collaborate on model…Happy LLM is an open-source project aimed at building large language models from scratch. It provides resources and tools for developers to contribute and collaborate on model…

AI Models & LLMs

See the Top 250 AI GitHub Repos

llama.cpp

AI GitHub Repos

ggml-org/llama.cpp is a GitHub repository focused on LLM inference in C/C++. It allows developers to contribute to its development and utilize its capabilities for machine…

AI Models & LLMs

View Tool

GitHub repository focused on LLM inference in C/C++. It allows developers to contribute to its development and utilize its capabilities for machine…ggml-org/llama.cpp is a GitHub repository focused on LLM inference in C/C++. It allows developers to contribute to its development and utilize its capabilities for machine…

FeaturedAI Models & LLMs

See the Top 250 AI GitHub Repos

Petals

AI GitHub Repos

Petals is an open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…

Coding & Development

Ranked #102 in AI GitHub Repos

View Tool

Open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…Petals is an open-source project that enables users to run large language models (LLMs) at home using a BitTorrent-style approach. It offers faster fine-tuning and inference…

Coding & Development

Ranked #102 in AI GitHub Repos

Train LLM from Scratch

AI GitHub Repos

Train LLM from Scratch provides a comprehensive guide to training your own large language model. It covers everything from data acquisition to text generation, making it an…

AI Models & LLMs

View Tool

Provides a comprehensive guide to training your own large language model. It covers everything from data acquisition to text generation, making it an…Train LLM from Scratch provides a comprehensive guide to training your own large language model. It covers everything from data acquisition to text generation, making it an…

AI Models & LLMs

See the Top 250 AI GitHub Repos

Ling

AI GitHub Repos

Ling is an open-source mixture-of-experts language model family from Ant Group's inclusionAI, released on GitHub under the MIT license. It ships in multiple sizes for general…

AI Models & LLMs

View Tool

Open-source mixture-of-experts language model family from Ant Group's inclusionAI, released on GitHub under the MIT license. It ships in multiple sizes for general…Ling is an open-source mixture-of-experts language model family from Ant Group's inclusionAI, released on GitHub under the MIT license. It ships in multiple sizes for general…

AI Models & LLMs

See the Top 250 AI GitHub Repos

FlashInfer

AI GitHub Repos

FlashInfer is a kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…

MLOps & Model Deployment

View Tool

Kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…FlashInfer is a kernel library designed for serving large language models (LLMs). It facilitates efficient deployment and management of LLMs, enabling developers to contribute and…

MLOps & Model Deployment

See the Top 250 AI GitHub Repos

OpenLLaMA

AI GitHub Repos

OpenLLaMA is a permissively licensed, open-source reproduction of Meta AI's LLaMA 7B model. Trained on the RedPajama dataset, it offers 3B, 7B, and 13B parameter versions,…

AI Models & LLMs

View Tool

Permissively licensed, open-source reproduction of Meta AI's LLaMA 7B model. Trained on the RedPajama dataset, it offers 3B, 7B, and 13B parameter versions,…OpenLLaMA is a permissively licensed, open-source reproduction of Meta AI's LLaMA 7B model. Trained on the RedPajama dataset, it offers 3B, 7B, and 13B parameter versions,…

AI Models & LLMs

See the Top 250 AI GitHub Repos

1 of 8