Description
The BERT (Bidirectional Encoder Representations from Transformers) project provides pre-trained models for natural language processing tasks. Among its offerings are multilingual models designed to handle a wide array of languages, enabling cross-lingual understanding and applications. Currently, two primary multilingual models are available: BERT-Base, Multilingual Cased (New, recommended) and BERT-Base, Multilingual Uncased (Orig, not recommended).
Read More
The BERT-Base, Multilingual Cased model supports 104 languages and features a 12-layer architecture with 768 hidden units, 12 attention heads, and 110 million parameters. This model is recommended due to its improved normalization for many languages, particularly those with non-Latin alphabets. When using this model, it is crucial to set `--do_lower_case=false` in the run scripts. The Multilingual Uncased model, while supporting 102 languages with the same architecture, is not recommended for new projects.
For Chinese language tasks, a dedicated BERT-Base, Chinese model is also available, supporting Simplified and Traditional Chinese. While the multilingual models include Chinese, a Chinese-only model may yield better results for Chinese-specific fine-tuning. The performance of these models has been evaluated on the XNLI dataset, a cross-lingual natural language inference task. Results indicate that while single-language models may outperform multilingual models for high-resource languages, the latter offer a practical solution for broad language coverage without the need to maintain numerous single-language models.
Fine-tuning the multilingual BERT models does not require significant API changes. However, updates to the `BasicTokenizer` may be necessary for Chinese character tokenization. The models can be integrated into workflows using provided scripts like `run_classifier.py`, which has been updated to support datasets like XNLI. The data sampling strategy for pre-training involved exponentially smoothed weighting of Wikipedia data to balance high-resource and low-resource languages, ensuring better representation for all. Tokenization uses a shared WordPiece vocabulary, with specific handling for CJK languages to address the absence of whitespace. The multilingual model intentionally omits language markers to facilitate zero-shot learning capabilities.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
BERT Multilingual Models Highlights
Supports 104 languages (Multilingual Cased)
Supports 102 languages (Multilingual Uncased)
12-layer transformer architecture
768 hidden units
12 attention heads
110 million parameters
Multilingual Cased model recommended for improved normalization
Zero-shot learning capability
Fine-tuning support for specific tasks
Pre-trained on Wikipedia data
Shared WordPiece vocabulary
Handles CJK languages with character tokenization
Open-source TensorFlow code available
Getting Started with BERT Multilingual Models
Access model: Download pre-trained multilingual BERT models.
Set up environment: Install TensorFlow and necessary dependencies.
Integrate via API: Load models and tokenizers using provided libraries.
Prepare data: Format your multilingual text data for training or inference.
Fine-tune: Adapt the model to specific downstream tasks like classification or question answering.
Run inference: Use the fine-tuned model to process new text data.
Evaluate performance: Assess model accuracy on relevant multilingual benchmarks.
BERT Multilingual Models's Use Cases
- Cross-lingual classification
- Multilingual sentiment analysis
- Zero-shot cross-lingual transfer
- Machine translation evaluation
- Multilingual question answering
- Cross-lingual information retrieval
- Natural language inference
