Executive Summary

Key Numbers

Source: NVIDIA and AI4Bharat, Hugging Face announcement (October 16, 2026).

AI4Bharat is an Indian language research lab based at IIT Madras. It has spent years assembling open corpora in Indian languages, and Mitesh Khapra of the lab has put the reason for that work into a single sentence. There is not enough data to go around in Indian languages. This announcement is about filling that gap, and the method is not collection.

According to the post NVIDIA and AI4Bharat put on Hugging Face on October 16, the new pretraining data covers all 22 languages listed as official in India's constitution. By the 2011 census those are the first languages of 97% of the country, roughly 1.2 billion people. The size is 2.8 trillion tokens as counted by the Gemma-4 tokenizer.

Those 2.8 trillion split into two parts. Text written in each language's own script accounts for 2,601.0B tokens and spans all 22 languages. Text written in the Roman alphabet accounts for 224.6B tokens and covers only 12 of them. Counted together, the 22 languages yield 34 language-script combinations. Roman script here does not mean text translated into English. It means an Indian language spelled out in Latin letters, the way people type Hindi on a phone keyboard. Native script likewise refers only to the writing system, and the announcement notes separately that this text was carried over from English as well.

The announcement also says what counts as one item in those totals. A document record can contain translations and script variants that branched from the same original. The figure is not a count of how many distinct English documents went in. One English source grew into several languages and two scripts, and that is how the token totals were filled.

Where the data came from is the heart of this release. The source is 15 subsets of Nemotron-CC-v2, the English web corpus NVIDIA curated, holding web documents, question-and-answer pairs, math, code and specialized content. The 2.8 trillion Indic tokens are what a translation model produced by carrying those English documents across one whole document at a time. The authors do not hide this. Their stated advantage is that every record traces back to a single source document, which makes the corpus auditable.

One point about timing should be clear. The dataset is named Nemotron-pretraining-Indic-v1, and the announcement says it is releasing soon on Hugging Face. As of October 18 it cannot be downloaded. What is public right now is the post describing the method, plus a GitHub plugin containing the translation, evaluation and rewriting prompts.

The 22 languages were not handled in one way. The 12 languages with some accumulated translation resources were split off from the other 10 and run through separate workflows. Both share one choice: they translate a whole document rather than a sentence, so that surrounding context survives.

The row where the two workflows diverge most is what happens below the bar. The rewriting loop exists only in the 12-language pipeline. The 10-language pipeline applies the same 75-point threshold but simply discards what falls short, and it produces no Roman-script text. The fewer translation resources a language has, in other words, the simpler its pipeline became. The announcement's own note that the 10 languages need more native-speaker validation points at the same spot.

The order in which Roman-script text is produced was a choice too. The team could have generated Romanized Indic text straight from English. Going into the native script first and converting the letters afterward gave better results, the post says: less invented content, and fewer scraps of untranslated English left behind.

The translation prompt is written to separate what should be carried over from what should be left alone. Prose gets translated; code, equations, URLs and formatting stay as they are. The whole pipeline runs stage by stage on NeMo Data Designer, NVIDIA's synthetic data generation framework, and the plugin used here, holding the translation and evaluation prompts, the rewriting step, long-document splitting and script conversion, is published on GitHub.

This is the part a data quality practitioner will sit with longest. A finished translation does not go straight into the corpus; it gets scored first. The scorecard is modeled on MQM, the framework used to assess translation quality, and it rates meaning, fluency, terminology, style and format out of 100. The bar is 75. The announcement attaches a caveat to that number: it is a rubric score, not an accuracy percentage. It does not say that 75% of the translation is correct. It says the five categories of the scorecard add up to 75.

In ordinary corpus building, whatever falls below the bar is thrown out. This pipeline handled it differently. A document under 75 receives the reasons it lost points as feedback, is rewritten, and is scored again on the same criteria. As the previous section showed, that loop lives in the 12-language workflow. What it brought back, counting native and Roman scripts together, is 438.4 billion tokens. Since that share is already inside the 2.8 trillion, it should not be read as the repair work having made the corpus bigger. The right reading is that without the repair step, the 2.8 trillion would have stopped somewhere around 2.4 trillion.

The authors know the grader is a machine. So they had people review a sample, and they seeded documents with deliberate errors to see whether the scoring prompt caught them. What came back from that was used to revise the prompt and the checklist.

The limitations are written into the announcement as well. Quality judgment is largely automatic, and how far human review extends varies by language. The authors list it among their limitations that in the 12-language workflow the model doing the translating also does the scoring. A model grades what it wrote. Alongside that sits the note that the 10-language workflow needs more native-speaker validation. Errors may remain in what was kept. Deduplication removes records that came from the same original, so it does not group texts that are merely similar in meaning. A data quality document that spells out the holes in its own pipeline this plainly is not common.

Whether the data is any good shows up when a model eats it. NVIDIA ran continued pretraining on Nemotron 3.5 Super VL with a budget of 100 billion tokens, of which 10%, or 10 billion tokens, was Indic. Not all of that 10 billion is this corpus. It is a mixture of the corpus's 12-language portion with text originally written in Indian languages, and the remaining 90% kept the ratios of the earlier training blend.

The scores went up. Five Indic knowledge and reasoning benchmarks, MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy and Indic GSM8K, all improved, by between 4.98 and 9.82 percentage points. The authors attach two caveats. These figures compare checkpoints before and after the continued-pretraining stage, so they do not isolate what the synthetic corpus alone contributed. And the benchmarks differ in which languages they cover. GSM8K was measured without Odia, because that evaluation exceeded the serving context limit.

Separately from that, the announcement also sets the finished Nemotron 3.5 Super VL against the previous generation, Nemotron 3 Super. That is not a before-and-after within one stage but a comparison between models that have been through post-training. Here, five of six tasks spanning knowledge, reasoning, math, code and translation improved. The translation task, WMT24++, is measured in chrF rather than percentage points. The four axes of IndiVibe, which looks at language quality, also improved: helpfulness, fluency, fidelity to the language and script, and appropriate length. IndiVibe is a benchmark of Indic conversational prompts released by Sarvam AI, and the values the announcement reports are preliminary conditional preference scores, not head-to-head win rates.

The axis that fell is the remaining one of those six. On IFEval 1.5, the instruction-following evaluation, the score dropped by 2.04 points. That test looks at whether a model holds to the format and the conditions a user asked for. The announcement does not explain why it fell.

The yardstick these scores are measured with also comes from the side that made the data. AI4Bharat is building the other half of the evaluation, a benchmark collection called Indic-LLM-Eval-Suite, with NVIDIA's support, and the announcement introduces it alongside the data. It is an open tool, so outsiders can measure with the same yardstick, but that cannot be left out of how the scores are read. On that point, the fact that IndiVibe is a yardstick built by an outside organization works as a counterweight.

Four of the five are multiple-choice tests of knowledge and reasoning, and the fifth is math. Knowledge that sat in English originals was carried into Indic sentences and fed to the model, so a rising score on a test that asks about that knowledge in an Indian language is evidence that the pipeline worked as designed. How much of that knowledge reflects the life and local context of people in India is a different question, and these five scores cannot answer it.

This is too close to the Korean situation to read as news about an Indic corpus and move on. Open training data in Korean is far scarcer than in English, and the fastest route to filling that gap is the same one: run English data through a translator and make up the volume. What is different about this announcement is that it did not hide the route but wrote it down as a pipeline and published it. That gives practitioners something to examine.

The most instructive part is the design that pins the bar to a single number and decides in advance what happens to whatever falls below it. Because there is a line at 75, the volume that fell under it can be counted, and the volume rescued by rewriting can be written down separately as 438.4 billion tokens. In a pipeline with no line, neither can be counted. What we usually see in the field is the opposite: the quality standard lives in somebody's head, and so the data that was discarded and the data that was fixed leave no record.

For a team filling out its data through translation, at least these four things should be written down somewhere.

• • What the passing score is and what that score measures. The number 75 itself is not worth borrowing. What matters is that the score is broken out across categories such as meaning, fluency, terminology, style and format, and that if a model does the scoring, there is a procedure for people to review that scoring on a sample
• • Whether text below the bar is dropped or repaired, and whether the repaired volume is counted separately. Rewritten data is not the same kind of thing as data that passed the first time. Mix them together and when the model starts behaving oddly later, there is no way to tell which half to suspect
• • Whether a translation can be traced back to its source document. This is why the pipeline can claim auditability. Synthetic data with the path back to the original cut off leaves no way to find where a problem came from
• • Whether records branched from the same original straddle both training and evaluation. In a pipeline where one source grows into several languages and two scripts, a sibling of the text held out for evaluation easily stays on the training side. The announcement makes the same point, advising that records from the same source be grouped across languages and scripts when building held-out splits

How did the authors say this corpus should be used? Do not train on it alone; obtain a reasonable amount of text originally written in Indian languages, mix it in, and then evaluate that blend for the use you have in mind. They add that generation and acceptance should be kept apart. The published plugin returns candidates and scores, and which of those go into a training corpus is settled by the filtering policy of whoever uses it. That separation between scoring something and accepting it transfers directly to work that has nothing to do with translation.

The question that remains is not one a score settles. If the Indic speech of a model 1.2 billion people will use is speech carried over from English, then the world written on the English web rides along with it. Benchmark scores do not catch that. Knowledge that was carried over, asked about in the language it was carried into, is going to be answered correctly. The same illusion shows up when Korean models are evaluated. If translated data goes in and the score rises on evaluation items that were themselves built by translation, that rise is not evidence that the Korean got better. It is evidence that the pipeline ran.

Thank you for reading this far. The facts here were checked against the announcement NVIDIA and AI4Bharat posted on Hugging Face on October 16, and this was written without sight of the dataset itself, which is not yet public. Once the actual files go up, we intend to check the per-language volumes and the script breakdown again. If you have an example of setting a quality bar while filling out training data through translation, we would be glad to hear it.

Pebblous Data Communication Team
October 18, 2026

• 1.NVIDIA · AI4Bharat (IIT Madras). (2026). "Scaling Indic Pretraining." Hugging Face Blog, October 16, 2026. (Primary source for sections 1 through 4: token counts, language coverage, the translation, scoring and rewriting pipeline, the evaluation results and the limitations the authors set down)
• 2.NVIDIA NeMo. (2026). "DataDesignerPlugins — Nemotron Bharat translation plugin." GitHub. (Section 2: the published plugin holding the translation and evaluation prompts, the rewriting step, long-document splitting and script conversion)
• 3.Unite.AI. (2026). "NVIDIA unveils 2.8T-token Indic pretraining corpus and open pipeline." (Reporting on the announcement. The figures were cross-checked against the primary source)