Model · EMNLP 2023
GlotLID
Language identification with support for more than 2,000 labels.
Overview
GlotLID is an open-source language identification (LID) model built on fastText. Version 3 supports 2,102 labels, each an ISO 639-3 code paired with an ISO 15924 script, and adds “zxx” labels for non-linguistic web noise and “und” labels for undetermined text. It is the language filter behind GlotCC and is widely used for cleaning multilingual web corpora.
Features
- Language identification for 2,000+ language–script labels
- Sentence vectors for many languages
- Restrict predictions to a subset of languages without retraining
- Explicit noise (zxx) and undetermined (und) labels