01 Latest · quipu-moe-1B-A149M
A 1B mixture of experts that uses 149M parameters per token.
Pretrained from scratch on one rented RTX 5090. Cost from the spend ledger, including the A/B tests, the chat fine-tune and evaluation.
What changed from 114M
16 layers, each with one shared expert and 64 small routed experts, of which every token uses 4. Routing is balanced without an auxiliary loss (Quantile Balancing). A new 49,152-token tokenizer, trained for this model, keeps code indentation intact and covers ten languages.
Every design switch was A/B tested at 200M tokens before the full run, and the losers are published too. Muon beat AdamW clearly: 3.32 vs 4.06 loss, against seed noise of 0.03.
Data: 60% permissively licensed code, 28% English, 12% Indonesian, Malay, Chinese, Japanese, Korean, Tamil, Hindi and Urdu, each document checked with my own language-ID model. HumanEval and MBPP were filtered out of the training data.
Bits per byte at the end of pretraining
Lower is better, comparable across tokenizers. Held-out text per language, and held-out code.
02 First model · quipu-114m
A 114M-parameter decoder, sized for one laptop.
Every number here was measured on the machine that trained it, an RTX 5060 Laptop GPU with 8 GB.
Architecture
Decoder-only transformer in the modern style: RMSNorm, rotary position embeddings, SwiGLU feed-forward, and grouped-query attention with 12 query heads sharing 4 key/value heads. Input and output embeddings are tied, and the context is 1,024 tokens.
Short context by design. Long-range recall is meant to come from memory rather than a longer attention window.
03 Results · quipu-114m
45 hours of training: what it learned.
All 5,722 steps ran unattended over a weekend: no restarts, no skipped steps, no data repeated. Validation loss on held-out FineWeb-Edu text, measured every 100 steps:
Watch it learn
Same prompt, greedy decoding, at each saved milestone.
It sounds right but often isn't.
The final model says the printing press was invented "in 1848 by the French printer Pierre-Louis Leclerc". That is normal for a 114M base model: it writes fluent sentences, but the facts are often wrong, and it still repeats itself under greedy decoding.
The code loss of 0.97 flatters it.
GPT-2's tokenizer splits indentation into single spaces: 36.5% of the code validation tokens are pure whitespace, against 2.0% for text. They are easy to predict, and greedy code output collapses into runs of spaces. That measurement is why the next model gets a code-aware tokenizer.