AI
NVIDIA and AI4Bharat plan a large synthetic corpus for Indian-language models
NVIDIA, IIT Madras and AI4Bharat describe a planned release of roughly 2.8 trillion tokens across 22 Indian languages, alongside an open pipeline for translation, quality assessment, repair and script conversion. The corpus includes native-script and romanized data with traceable source documents. A related continued-pretraining stage improved five reported Indic benchmarks, but the experiment combined synthetic and natural data, so it does not isolate the synthetic corpus’s effect. The authors also caution that automated checks and translation cannot replace native-language review or naturally authored local content.
Editorial summary · Publisher article
What matters
- The corpus covers 34 language-script configurations and draws on web, mathematics, code and question-answer material.
- Accepted translations generally require a rubric score of at least 75 out of 100.
- The reported continued-pretraining gains range from 4.98 to 9.82 percentage points.
- The release is intended to support reproducible mixture studies and adaptation to new domains.
Context & caveats
- The source is a publisher-authored account of the project, and the corpus release is described as forthcoming.
Why this made the edition
The long technical account explains the corpus, generation pipeline, evaluation design, reproducibility options and limitations, including where the authors' benchmark evidence cannot isolate causation.
Source material