Description
ERNIE-5.5-Flash-Base-BF16 is a sophisticated sparse Mixture-of-Experts (MoE) language model developed by Baidu. It features a substantial 280 billion total parameters, with 14 billion activated per token, making it highly efficient for complex language tasks. The model was pre-trained from scratch on an extensive dataset of 33.5 trillion tokens, utilizing a training context of up to 128,000 tokens. For fine-tuning and inference, it supports an impressive 1 million-token context length, enabling deep understanding and generation of long-form content.
Read More
Designed with efficiency as a core principle, ERNIE 5.5 Flash revisits every component, from attention mechanisms and residual connections to MoE layers and the output head. This holistic approach ensures optimal performance while managing compute, KV cache, and long-context costs. The pre-training data emphasizes capabilities crucial for advanced applications, such as repository-scale code understanding and long-context comprehension, allowing developers to build upon its general capabilities for diverse use cases.
ERNIE-5.5-Flash-Base-BF16 represents the checkpoint at the end of pre-training, serving as a foundational model before any mid- or post-training stages. This makes it an ideal starting point for researchers and developers engaged in domain adaptation, instruction tuning, and reinforcement learning. The model has demonstrated strong results across a unified evaluation pipeline, excelling in benchmarks related to knowledge, reasoning, mathematics, code, multilingual tasks, and long-context understanding.
The architecture incorporates a 280B-A14B hybrid-attention MoE with 43 layers, including 1 dense FFN and 42 Multi-Router Latent MoE layers. It features 37 Heavily Compressed Attention (HCA) layers and 6 Multi-head Latent Attention (MLA + DSA) layers with sparse attention. The model utilizes 512 routed experts (top-10) plus 1 shared expert per MoE layer. Innovations like Low-Rank Gated Attention, Virtual Head Attention (VHA), Multi-Router Latent MoE, ECHO, Manifold-constrained Hyper-Connections (mHC), and MultiMax Output Head contribute to its advanced capabilities and efficiency. The tokenizer supports approximately 200,000 entries and tokenizes numbers right-to-left in groups of up to three digits.
Deployment is supported with SGLang and vLLM, allowing for production serving with context lengths up to 1,048,576 tokens. The model is available under the ERNIE 5.5 Flash Base Model License Agreement v1.0.
- CategoryAI Models & LLMs
- DeploymentCloud
- PlatformsWeb
ERNIE-5.5-Flash-Base-BF16 Highlights
280 billion total parameters
14 billion activated parameters per token
Pre-trained on 33.5 trillion tokens
Training context up to 128,000 tokens
Fine-tuning and inference support 1 million-token context
Efficient sparse Mixture-of-Experts (MoE) architecture
Hybrid-attention MoE with 43 layers
37 Heavily Compressed Attention (HCA) layers
6 Multi-head Latent Attention (MLA + DSA) layers
512 routed experts (top-10) plus 1 shared expert per MoE layer
Low-Rank Gated Attention and Virtual Head Attention (VHA)
Multi-Router Latent MoE for accurate expert selection
ECHO for multi-horizon loss and speculative decoding
Manifold-constrained Hyper-Connections (mHC) for signal propagation
MultiMax Output Head for sharpened vocabulary distribution
Approximately 200,000-entry BPE tokenizer
Support for SGLang and vLLM for production serving
Getting Started with ERNIE-5.5-Flash-Base-BF16
Access Model Page: Navigate to the ERNIE-5.5-Flash-Base-BF16 page on Hugging Face.
Explore Files: Review the model files and documentation available on the page.
Download Model: Download the ERNIE-5.5-Flash-Base-BF16 model checkpoint.
Install vLLM and Plugin: Install vLLM and the ERNIE 5.5 plugin for deployment.
Launch Server: Use SGLang or vLLM to launch the model server.
Send Requests: Send requests to the completions endpoint using tools like curl for inference.
ERNIE-5.5-Flash-Base-BF16's Use Cases
- Text Generation
- Domain Adaptation
- Instruction Tuning
- Reinforcement Learning Research
- Code Understanding
- Long-Context Comprehension
- Multilingual Applications
- Reasoning Tasks
- Agentic Applications









