01 Data v3

Ten billion tokens of pretraining data, built for $2.51.

The next pretraining batch: 10,000,003,101 tokens from 42 open, licence-checked sources, built on Modal CPUs. Benchmark problems (HumanEval and MBPP) were filtered out, as before.

10.0Btokens (10,000,003,101)
$2.51to build, on Modal CPUs
42open, licence-checked sources
45%code from starcoderdata

What's in it

Mostly code and educational text, with maths, reference text, nine languages from FineWeb-2, and a small slice of the verified teacher answers from distillation v2.

code + verified answerstext, maths, languages
Codestarcoderdata45%
Educational web textFineWeb-Edu20%
Nine languagesFineWeb-210%
Maths7%
Stack Exchangedump of March 20235%
Older codegithub-code-clean5%
ReferenceWikipedia, peS2o, arXiv4%
Code-related web3%
Verified answersdistillation v21%

02 Data · quipu-114m

Three billion tokens: mostly text, one-fifth code.

80% FineWeb-Edu, 20% code from github-code-clean. Only permissive licences are kept: MIT, Apache-2.0, BSD, ISC, CC0 and Unlicense. Minified files, vendored directories and files over 16k tokens are filtered out. That filter alone removed as many tokens as the entire code budget.

Educational web text · 80%Code · 20%

HTML is capped at 10% so markup doesn't crowd out the languages the model should actually learn to write.

NextEngineering