Description
AliceAI-Foundation-80B-A3B-Base is a foundational language model developed by Yandex, hosted on Hugging Face. It is designed with a hybrid architecture incorporating Mixture of Experts (MoE) layers, and it boasts a total of 80 billion parameters, with 3 billion active for each token. The model supports a context length of up to 262,144 tokens, making it suitable for handling extensive text inputs.
Read More
The model was trained from scratch, with a newly assembled training corpus and carefully selected architecture and hyperparameters. It is particularly strong in tasks involving factual knowledge in the Russian language, outperforming larger open-source models in this area. The model's performance is validated through benchmarks like WikiWebFacts and HardMultiQA, which are focused on Russian contexts.
AliceAI-Foundation-80B-A3B-Base is an autoregressive language model, currently in the pre-training stage. It features a hidden state size of 2048, a vocabulary size of 129,024, and consists of 48 layers. The model's architecture includes 12 blocks of 3 KDA to MoE layers followed by 1 Gated Attention to MoE layer.
The model's capabilities are demonstrated through various benchmarks, where it shows competitive results in mathematics, programming, and reasoning tasks. It is particularly effective in factual knowledge tasks in Russian, as evidenced by its performance in the WikiWebFacts and HardMultiQA benchmarks.
AliceAI-Foundation-80B-A3B-Base is available for use through the Transformers library, with a reference version of 5.16.1. It can also be deployed using vLLM, requiring Docker and NVIDIA Container Toolkit for execution. The model is open for further fine-tuning and adaptation to specific tasks, with support for LoRA fine-tuning provided in the repository.
This model is primarily intended for research, experimentation, and further development. It is not a ready-to-use solution for end-user products and services. Users are advised to conduct their own testing and implement necessary fine-tuning and control measures before deploying the model in production environments.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
AliceAI-Foundation-80B-A3B-Base Highlights
80 billion parameters
3 billion active parameters per token
Hybrid architecture with MoE layers
Supports context length up to 262,144 tokens
Strong performance in Russian factual knowledge tasks
Pre-trained from scratch
Available through Transformers and vLLM
Supports LoRA fine-tuning
Getting Started with AliceAI-Foundation-80B-A3B-Base
Access model page: Visit Hugging Face
Authenticate: Log in to your account
Configure: Set up the model parameters
Send prompts: Input your text queries
Fine-tune: Adapt the model to specific tasks
AliceAI-Foundation-80B-A3B-Base's Use Cases
- Factual Knowledge Retrieval
- Mathematical Problem Solving
- Programming Assistance
- Language Understanding
- Research and Development
