01 Design decisions

Design decisions and why I made them.

Each step below is a question I had to answer while building Quipu. The amber branch is what I picked; the grey ones are what I turned down, with the measurement or rule that decided it. Where an A/B test was run, the rule for winning was written down before the result came in.

01

Starting point

  1. Base modelFine-tune an existing model, or train one from scratch?

    chosenTrain from scratch

    Own weights, own tokenizer, own data pipeline. Every number on this site comes from a run I set up, so I can explain all of it.

    No fine-tuned base anywhere in Quipu
    not chosenFine-tune an open model

    Much faster to a good chatbot, but the interesting parts (data, tokenizer, pretraining) would belong to someone else.

  2. First modelHow big should the first model be?

    chosen114M parameters, 3B tokens

    Sized for what the 8 GB laptop GPU could finish over one weekend. The point was to prove the whole pipeline before spending money.

    45 h 22 m, zero restarts, no rented compute
    not chosen350M first

    Would have been a better model, but about 8 days of laptop GPU with no proof the pipeline worked.

    ~8 days estimated
  3. ContextHow long should the context window be?

    chosen1,024 tokens

    On an 8 GB card a long window's key/value cache eats the memory the weights need. Long-range recall is meant to come from retrieval instead.

    The demo stretches it to 4,096 with RoPE scaling
    not chosenA long window (32k+)

    Does not fit next to the weights on the training GPU.

02

Tokenizer

  1. TokenizerKeep GPT-2's tokenizer for the second model?

    chosenTrain a new 49,152-token tokenizer

    GPT-2's tokenizer splits code indentation into single spaces, so the first model's code loss looked better than it was. The new one keeps indentation together and covers ten languages.

    Code: 3.79 characters per token, vs 2.32 for GPT-2
    not chosenKeep GPT-2's

    No extra work, but 36.5% of code tokens were pure whitespace, and greedy code output collapsed into runs of spaces.

    36.5% whitespace tokens in code, 2.0% in text
03

Architecture and optimiser

  1. Model shapeDense model or mixture of experts for the 1B run?

    chosenMixture of experts: 998M total, 149M per token

    16 layers, each with one shared expert and 64 small routed experts, of which each token uses 4. A 1B model's capacity at roughly the per-token cost of a 150M one, which kept the run under $20.

    $18.75 total compute
    not chosenDense 1B

    Every token pays for every parameter: far fewer training tokens for the same money.

  2. Expert routingHow should the experts be kept evenly used?

    chosenQuantile Balancing

    Balances the routing directly, with no extra loss term pulling against the language-modelling loss.

    No auxiliary loss to tune
    not chosenAuxiliary balancing loss

    The common approach, but it adds a weight to tune and can trade model quality for balance.

  3. OptimiserMuon or AdamW?

    chosenMuon

    A/B tested at 200M tokens before the full run. The gap was more than 20 times the seed noise.

    Loss 3.32 vs 4.06, seed noise 0.03
    not chosenAdamW

    The standard choice, and clearly worse here.

    Loss 4.06
  4. Attention residualsAdd Block AttnRes?

    chosenLeave it out

    It had better loss, but the rule written down before the test said: adopt only if it fits in memory and is at most 25% slower at full size. It failed the memory check.

    Full-size probe: 31.87 GB, over the 30.98 GB limit
    not chosenUse Block AttnRes

    Loss 3.03 against 3.32 in the A/B, but 15.5% slower, and too big at full size.

    15.5% slower
  5. ActivationSiTU-GLU or SwiGLU?

    chosenSwiGLU

    SiTU-GLU won the A/B, but by less than the same money spent on more training tokens was expected to buy. This override is documented in the model card.

    SiTU-GLU gain 0.047; tokens worth ~0.10
    not chosenSiTU-GLU

    A real but small gain that did not pay for itself.

  6. PrecisionTrain in FP8 to go faster?

    chosenStay in bf16

    FP8 only helps if it is clearly faster. On this GPU there is no FP8 grouped matmul for the experts, and the measured result was slower.

    FP8 at 0.79–0.89x the speed of bf16; rule needed 1.2x
    not chosenFP8

    Saves memory and can be faster on the right hardware, but not here.

04

Hardware

  1. ComputeWhich GPU for the 1B pretraining run?

    chosenOne rented RTX 5090

    Measured as the most tokens per dollar for this small model. Results were copied back and the machine destroyed right after.

    ~$0.52/h, 44.3k tokens/s
    not chosenH100 / H200 / B200

    Faster per GPU but much more per hour for a model this size.

    $1.95 / $2.95 / $7.81 per hour (rental prices, 30 Sep)
    not chosenGB10 desktop boxes

    Cheap, but roughly the same speed as the laptop.

    29.5 TFLOPS
05

Data

  1. Code licencesWhich code is allowed in the training data?

    chosenPermissive licences only

    MIT, Apache-2.0, BSD, ISC, CC0 and Unlicense. Minified files, vendored folders and huge files are dropped, and HTML is capped at 10% so markup doesn't crowd out real languages.

    HumanEval and MBPP filtered out of training data
    not chosenAny public code

    More data, but unclear rights to train on it.

  2. LanguagesEnglish only, or more languages?

    chosenEnglish plus nine Asian languages

    Indonesian, Malay, Chinese, Japanese, Korean, Tamil, Hindi and Urdu. Each document is checked by my own language-ID model before it goes in.

    Mix: 60% code, 28% English, 12% the other languages
    not chosenEnglish only

    Simpler, but the tokenizer and model would be poor at the languages around me.

  3. Data v3 sourcesWhich sources go into the 10B-token set?

    chosen42 open sources, licences checked

    Stack Exchange is used only from an older dump, because newer dumps forbid training language models. Large synthetic sets made by other companies' models were left out.

    Built for $2.51 on CPUs
    not chosenNewest dumps and synthetic sets

    Bigger and more recent, but with terms that do not allow this use.

06

Distillation

  1. Teacher modelWhich model should teach Quipu to code?

    chosenQwen2.5-Coder (Apache-2.0)

    An open-weight model whose licence allows training on its outputs.

    1.5B for v1, 7B for v2
    not chosenClosed API models

    Often stronger, but their terms forbid using their outputs to train other models.

  2. Quality checkHow do we know a teacher answer is good?

    chosenRun it against the problem's unit tests

    Every answer runs in a sandbox and only passing ones are kept. A bug where the harness shared variables with the answer was found and fixed, and every part was checked again.

    v2: 498,813 of 638,342 kept (78.1%)
    not chosenAsk another model to grade it

    Cheaper to set up, but a grade is an opinion; a passing test is a fact.

  3. Scaling upKeep the small teacher for v2?

    chosenMove to the 7B teacher

    The bigger teacher passes its own tests more often, so more of the compute ends up as kept answers.

    Pass rate 65.5% (1.5B) → 78.1% (7B)
    not chosenStay on 1.5B

    Cheaper per answer, but a third of answers were thrown away.

07

Scaling

  1. Next runsHow to keep training without a budget?

    chosenMonthly runs on free compute credits

    Multi-GPU training was built and verified on 2 real GPUs before asking for any bigger machine, and the expert dispatch was made 3.24x faster first so every credit goes further.

    113k → 366k tokens/s on a B200
    not chosenRent a big cluster now

    Fastest, but not affordable for a personal project.

NextRoadmap