NewA 10B-token pretraining set from 42 open sources

Small language models, trained from scratch on a tight budget.

Own weights, own tokenizer, own data pipeline, no fine-tuned base. It started on an 8 GB laptop GPU; the current model is a 1B mixture of experts that cost under $20 to train on one rented GPU.

fig. 01 · code mixquipu-114m
quipu-114m's code mix, recorded the Incan way: one cord per language, clusters of knots for tens (upper) and units (lower). Java's cord reads 3 and 5, for 35%.

00 On this site

What's on this site.

NextThe demo, recorded