GlotWeb

Overview

GlotWeb finds, validates and catalogues web pages written in minority languages. It aggregates search results from several engines, verifies each page with GlotLID, filters for quality while reducing religious-text bias, and publishes a browsable index — 47% of its links are in languages absent from major multilingual datasets.

Features

• Four-step pipeline: search, seed, crawl, clean
• Language validation with GlotLID
• Covers languages missing from FLORES-200, MADLAD-400 and Glot500
• Interactive demo for browsing sites per language

Links

• Paper
• Code
• Demo