00 On this site
What's on this site.
A 1B mixture of experts for $18.75.
998M parameters, 149M used per token, pretrained on one rented GPU. Plus the first model, trained on a laptop.
02 · distillationKeep only answers that pass their tests.
Training on 3,277 tested answers took MBPP from 1.9% to 7.4%. The next round has 498,813.
03 · dataTen billion tokens for $2.51.
42 open, licence-checked sources, benchmarks filtered out.
04 · engineeringBugs found before the long runs.
Multi-GPU training verified, expert dispatch 3.24x faster, and four bugs that would have failed quietly.
05 · decisionsWhy each choice was made.
Every major design choice as a decision tree: what was picked, what was rejected, and the number behind it.
06 · roadmapWhere it goes next.
Continued pretraining on free compute credits, then a chat fine-tune on all the verified answers.