Glot500

Overview

Glot500-m extends XLM-R-base from 104 to more than 500 languages through continued pretraining on Glot500-c, a corpus drawn from Glot2000-c (2,000+ languages). The model outperforms XLM-R-base on head and tail languages across sequence labelling, retrieval and classification tasks.

Features

• Glot500-m: masked language model for 500+ languages
• Glot500-c: training corpus with 30,000+ sentences per language
• Redistributable subset published on Hugging Face
• Evaluation code for head vs. tail languages

Links

• Paper
• Code
• Model
• Data