Description
ERNIE-5.5-Flash-Base is a sophisticated text generation model developed by Baidu, available on Hugging Face. It is an efficient sparse Mixture-of-Experts (MoE) language model, featuring 280 billion total parameters with 14 billion activated per token. The model was pre-trained from scratch on an extensive dataset of 33.5 trillion tokens, utilizing a training context of up to 128K. For fine-tuning and inference, it supports an impressive 1M-token context length.
Read More
Designed with efficiency as a core objective, ERNIE 5.5 Flash revisits every component, from attention mechanisms and residual connections to MoE layers and the output head. This holistic approach considers activated compute, KV cache, and long-context costs as part of an integrated design. The pre-training data emphasizes capabilities that are challenging to build post-training, such as repository-scale code understanding and long-context comprehension. The training system was co-designed with the architecture to ensure efficient scaling.
ERNIE-5.5-Flash-Base represents the checkpoint at the end of pre-training, making it an ideal foundation for developers. It allows for domain adaptation, instruction tuning, and reinforcement learning research. The model demonstrates strong results across a unified evaluation pipeline, covering knowledge, reasoning, mathematics, code, multilingual tasks, and long-context benchmarks. Its performance is benchmarked against other large models like DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base, all evaluated under consistent conditions.
The architecture incorporates a 280B-A14B hybrid-attention MoE with 43 layers, including 1 dense FFN and 42 Multi-Router Latent MoE layers. It features 37 Heavily Compressed Attention (HCA) layers and 6 Multi-head Latent Attention layers with sparse attention (MLA + DSA). The model utilizes 512 routed experts (top-10) plus 1 shared expert per MoE layer. Key innovations include Hybrid MLA + HCA attention with layer-specific head dimensions, Low-Rank Gated Attention, Virtual Head Attention (VHA), Multi-Router Latent MoE, ECHO for future token prediction, Manifold-constrained Hyper-Connections (mHC), and a MultiMax Output Head. The tokenizer uses a ≈200K-entry BPE tokenizer that tokenizes numbers right-to-left in groups of up to three digits.
Deployment of ERNIE-5.5-Flash-Base is supported with SGLang and vLLM, enabling production serving with context lengths up to 1,048,576 tokens for long-context inference. The model is provided under the ERNIE 5.5 Flash Base Model License Agreement v1.0, with copyright held by Baidu.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
ERNIE-5.5-Flash-Base Highlights
280B total parameters with 14B activated per token
Pre-trained on 33.5T tokens
Supports 1M-token context for fine-tuning and inference
Efficient sparse Mixture-of-Experts (MoE) architecture
Hybrid MLA + HCA attention with layer-specific head dimensions
Low-Rank Gated Attention and Virtual Head Attention (VHA)
Multi-Router Latent MoE with 512 routed experts and 1 shared expert
ECHO for multi-horizon loss and speculative decoding
Manifold-constrained Hyper-Connections (mHC) for signal propagation
MultiMax Output Head for sharpened vocabulary distribution
≈200K-entry BPE tokenizer with right-to-left number tokenization
Supports deployment with SGLang and vLLM
Strong performance across knowledge, reasoning, math, code, multilingual, and long-context benchmarks
FP8 and BF16 precision weights available
Getting Started with ERNIE-5.5-Flash-Base
Access Model: Navigate to the ERNIE-5.5-Flash-Base page on Hugging Face.
Download Model: Choose between FP8 (safetensors) or BF16 (safetensors) precision for download.
Install Deployment Tools: Install vLLM and the ERNIE 5.5 plugin, or use the official SGLang prebuilt Docker image.
Launch Server: Start the vLLM or SGLang server to host the model.
Send Requests: Use curl or other API clients to send requests to the completions endpoint for text generation.
Fine-tune Model: Utilize the pre-trained checkpoint for domain adaptation, instruction tuning, or reinforcement learning research.
ERNIE-5.5-Flash-Base's Use Cases
- Text Generation
- Domain Adaptation
- Instruction Tuning
- Reinforcement Learning Research
- Code Generation and Understanding
- Long-Context Understanding
- Multilingual Applications
- Reasoning Tasks
