Description
OpenLLaMA represents a significant advancement in open-source large language models, offering a permissively licensed reproduction of Meta AI's LLaMA. Developed by openlm-research, this project aims to democratize access to powerful language models by providing accessible weights and training methodologies.
Read More
The project releases a series of models, including 3B, 7B, and 13B parameter versions, all trained on one trillion tokens. These models are designed to be drop-in replacements for the original LLaMA models in existing implementations. The project provides weights in both PyTorch format, compatible with the Hugging Face transformers library, and JAX format for use with the EasyLM framework.
OpenLLaMA's training methodology closely follows the original LLaMA paper, utilizing the same model architecture, context length, training steps, learning rate schedule, and optimizer. The key distinction lies in the datasets used: v1 models are trained on the RedPajama dataset, while v2 models incorporate a mixture of the Falcon refined-web dataset, the StarCoder dataset, and parts of the RedPajama dataset (Wikipedia, ArXiv, Books, StackExchange). This commitment to open datasets ensures transparency and reproducibility.
The project emphasizes evaluation and comparison, presenting results against LLaMA and GPT-J across a wide range of tasks using the lm-evaluation-harness. OpenLLaMA models demonstrate comparable, and in some cases superior, performance to their counterparts. The development team has also addressed potential issues, such as tokenizer configurations affecting code generation tasks, recommending v2 models for such applications.
OpenLLaMA is licensed under the Apache 2.0 license, promoting broad adoption and modification. The project is a collaborative effort, with contributions from Berkeley AI Research and support from Google's TPU Research Cloud program. This initiative empowers researchers and developers to build upon and innovate with state-of-the-art language models.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud, Self-hosted
- PlatformsWeb
OpenLLaMA's Core Features
Permissively licensed open-source reproduction of Meta AI's LLaMA
Available in 3B, 7B, and 13B parameter sizes
Trained on 1T tokens
Weights available in PyTorch and JAX formats
Compatible with Hugging Face transformers and EasyLM framework
Follows original LLaMA architecture and training hyperparameters
Utilizes open datasets like RedPajama, Falcon refined-web, and StarCoder
Comparable performance to original LLaMA and GPT-J
Apache 2.0 license for broad usage
Includes evaluation results and comparisons
Tokenizer designed to merge multiple empty spaces, similar to T5
Getting Started with OpenLLaMA
Access Model Weights: Download PyTorch or JAX weights from Hugging Face Hub or EasyLM repositories.
Set Up Environment: Install necessary libraries such as transformers, PyTorch, or JAX.
Load Tokenizer: Use LlamaTokenizer or AutoTokenizer with `use_fast=False` to avoid potential tokenization issues.
Load Model: Instantiate LlamaForCausalLM from Hugging Face transformers or the equivalent in EasyLM.
Integrate via API: Utilize the loaded model for text generation, inference, or fine-tuning tasks.
Evaluate Performance: Employ lm-evaluation-harness for benchmarking, ensuring correct tokenizer usage.
OpenLLaMA's Use Cases
- Text Generation
- Language Understanding
- Model Reproduction
- Research and Development
- Chatbot Development
- Code Generation
