01 Design decisions
Design decisions and why I made them.
Each step below is a question I had to answer while building Quipu. The amber branch is what I picked; the grey ones are what I turned down, with the measurement or rule that decided it. Where an A/B test was run, the rule for winning was written down before the result came in.
- From scratch
- 114M on a laptop
- Short context
- Own tokenizer
- Mixture of experts
- No auxiliary loss
- Muon
- No AttnRes
- SwiGLU
- bf16, not FP8
- One RTX 5090
- Permissive code only
- Ten languages
- Clean terms
- Open teacher
- Run the tests
- 7B for v2
- Free credits
Starting point
-
Base modelFine-tune an existing model, or train one from scratch?
chosenTrain from scratchOwn weights, own tokenizer, own data pipeline. Every number on this site comes from a run I set up, so I can explain all of it.
No fine-tuned base anywhere in Quipunot chosenFine-tune an open modelMuch faster to a good chatbot, but the interesting parts (data, tokenizer, pretraining) would belong to someone else.
-
First modelHow big should the first model be?
chosen114M parameters, 3B tokensSized for what the 8 GB laptop GPU could finish over one weekend. The point was to prove the whole pipeline before spending money.
45 h 22 m, zero restarts, no rented computenot chosen350M firstWould have been a better model, but about 8 days of laptop GPU with no proof the pipeline worked.
~8 days estimated -
ContextHow long should the context window be?
chosen1,024 tokensOn an 8 GB card a long window's key/value cache eats the memory the weights need. Long-range recall is meant to come from retrieval instead.
The demo stretches it to 4,096 with RoPE scalingnot chosenA long window (32k+)Does not fit next to the weights on the training GPU.
Tokenizer
-
TokenizerKeep GPT-2's tokenizer for the second model?
chosenTrain a new 49,152-token tokenizerGPT-2's tokenizer splits code indentation into single spaces, so the first model's code loss looked better than it was. The new one keeps indentation together and covers ten languages.
Code: 3.79 characters per token, vs 2.32 for GPT-2not chosenKeep GPT-2'sNo extra work, but 36.5% of code tokens were pure whitespace, and greedy code output collapsed into runs of spaces.
36.5% whitespace tokens in code, 2.0% in text
Architecture and optimiser
-
Model shapeDense model or mixture of experts for the 1B run?
chosenMixture of experts: 998M total, 149M per token16 layers, each with one shared expert and 64 small routed experts, of which each token uses 4. A 1B model's capacity at roughly the per-token cost of a 150M one, which kept the run under $20.
$18.75 total computenot chosenDense 1BEvery token pays for every parameter: far fewer training tokens for the same money.
-
Expert routingHow should the experts be kept evenly used?
chosenQuantile BalancingBalances the routing directly, with no extra loss term pulling against the language-modelling loss.
No auxiliary loss to tunenot chosenAuxiliary balancing lossThe common approach, but it adds a weight to tune and can trade model quality for balance.
-
OptimiserMuon or AdamW?
chosenMuonA/B tested at 200M tokens before the full run. The gap was more than 20 times the seed noise.
Loss 3.32 vs 4.06, seed noise 0.03not chosenAdamWThe standard choice, and clearly worse here.
Loss 4.06 -
Attention residualsAdd Block AttnRes?
chosenLeave it outIt had better loss, but the rule written down before the test said: adopt only if it fits in memory and is at most 25% slower at full size. It failed the memory check.
Full-size probe: 31.87 GB, over the 30.98 GB limitnot chosenUse Block AttnResLoss 3.03 against 3.32 in the A/B, but 15.5% slower, and too big at full size.
15.5% slower -
ActivationSiTU-GLU or SwiGLU?
chosenSwiGLUSiTU-GLU won the A/B, but by less than the same money spent on more training tokens was expected to buy. This override is documented in the model card.
SiTU-GLU gain 0.047; tokens worth ~0.10not chosenSiTU-GLUA real but small gain that did not pay for itself.
-
PrecisionTrain in FP8 to go faster?
chosenStay in bf16FP8 only helps if it is clearly faster. On this GPU there is no FP8 grouped matmul for the experts, and the measured result was slower.
FP8 at 0.79–0.89x the speed of bf16; rule needed 1.2xnot chosenFP8Saves memory and can be faster on the right hardware, but not here.
Hardware
-
ComputeWhich GPU for the 1B pretraining run?
chosenOne rented RTX 5090Measured as the most tokens per dollar for this small model. Results were copied back and the machine destroyed right after.
~$0.52/h, 44.3k tokens/snot chosenH100 / H200 / B200Faster per GPU but much more per hour for a model this size.
$1.95 / $2.95 / $7.81 per hour (rental prices, 30 Sep)not chosenGB10 desktop boxesCheap, but roughly the same speed as the laptop.
29.5 TFLOPS
Data
-
Code licencesWhich code is allowed in the training data?
chosenPermissive licences onlyMIT, Apache-2.0, BSD, ISC, CC0 and Unlicense. Minified files, vendored folders and huge files are dropped, and HTML is capped at 10% so markup doesn't crowd out real languages.
HumanEval and MBPP filtered out of training datanot chosenAny public codeMore data, but unclear rights to train on it.
-
LanguagesEnglish only, or more languages?
chosenEnglish plus nine Asian languagesIndonesian, Malay, Chinese, Japanese, Korean, Tamil, Hindi and Urdu. Each document is checked by my own language-ID model before it goes in.
Mix: 60% code, 28% English, 12% the other languagesnot chosenEnglish onlySimpler, but the tokenizer and model would be poor at the languages around me.
-
Data v3 sourcesWhich sources go into the 10B-token set?
chosen42 open sources, licences checkedStack Exchange is used only from an older dump, because newer dumps forbid training language models. Large synthetic sets made by other companies' models were left out.
Built for $2.51 on CPUsnot chosenNewest dumps and synthetic setsBigger and more recent, but with terms that do not allow this use.
Distillation
-
Teacher modelWhich model should teach Quipu to code?
chosenQwen2.5-Coder (Apache-2.0)An open-weight model whose licence allows training on its outputs.
1.5B for v1, 7B for v2not chosenClosed API modelsOften stronger, but their terms forbid using their outputs to train other models.
-
Quality checkHow do we know a teacher answer is good?
chosenRun it against the problem's unit testsEvery answer runs in a sandbox and only passing ones are kept. A bug where the harness shared variables with the answer was found and fixed, and every part was checked again.
v2: 498,813 of 638,342 kept (78.1%)not chosenAsk another model to grade itCheaper to set up, but a grade is an opinion; a passing test is a fact.
-
Scaling upKeep the small teacher for v2?
chosenMove to the 7B teacherThe bigger teacher passes its own tests more often, so more of the compute ends up as kept answers.
Pass rate 65.5% (1.5B) → 78.1% (7B)not chosenStay on 1.5BCheaper per answer, but a third of answers were thrown away.
Scaling
-
Next runsHow to keep training without a budget?
chosenMonthly runs on free compute creditsMulti-GPU training was built and verified on 2 real GPUs before asking for any bigger machine, and the expert dispatch was made 3.24x faster first so every credit goes further.
113k → 366k tokens/s on a B200not chosenRent a big cluster nowFastest, but not affordable for a personal project.