Description
LIDstral Arabic is a sophisticated language and dialect classifier designed for text written in Arabic script. It is capable of identifying Modern Standard Arabic, a wide range of Arabic dialects, and non-Arabic languages that utilize the same script. This capability is crucial for distinguishing languages that share a script, preventing misclassification based solely on script appearance.
Read More
The model employs a combination of a fastText one-vs-all (OVA) classifier, an MLP stacker, and per-class isotonic calibration. The fastText OVA classifier generates independent scores for each class, maintaining evidence for competing languages and dialects. The MLP stacker, a 64-unit LayerNorm MLP, maps these scores to probabilities across classes, utilizing 144 features derived from raw scores, prior corrections, confidence margins, entropy, and regional summaries.
Per-class isotonic regressors are fitted on development predictions to calibrate the MLP probabilities, which are then renormalized across classes. This calibration process can alter both the confidence and the predicted label. The full pipeline requires three files: lid_ova_full.bin, mlp_stacker_ln.pt, and isotonic_calibration.pkl. The classify.py script runs the pipeline on CPU, including preprocessing and feature extraction, providing a predicted label and calibrated probabilities over all 51 classes for each input.
LIDstral Arabic has been evaluated on 84,870 Arabic examples from various benchmarks, achieving an 88.67% F1 score on Moroccan Darija, significantly outperforming other models like LahjatBERT ALDi CL and GlotLID v3. However, the model always assigns one of its 51 supported classes, which may lead to misclassification of unsupported languages or mixed-language text.
The model is licensed under the Apache 2.0 License, and users must ensure that their use does not infringe on third-party rights.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
LIDstral Arabic Highlights
Language and dialect classification
Modern Standard Arabic identification
Arabic dialect identification
Non-Arabic language distinction
fastText OVA classifier
MLP stacker with LayerNorm
Per-class isotonic calibration
51 supported classes
Getting Started with LIDstral Arabic
Access model: Download repository
Authenticate: Set up environment
Set up environment: Install dependencies
Integrate via API: Use classify.py script
Optimize: Calibrate probabilities
LIDstral Arabic's Use Cases
- Arabic Text Classification
- Dialect Identification
- Non-Arabic Language Detection
- Multilingual Text Analysis
- Language Distinction
