01 Latest · quipu-moe-1B-A149M

A 1B mixture of experts that uses 149M parameters per token.

Pretrained from scratch on one rented RTX 5090. Cost from the spend ledger, including the A/B tests, the chat fine-tune and evaluation.

998M / 149Mparameters, total / active
4.26Btraining tokens
44.3ktokens / second, bf16
$18.75total compute cost

What changed from 114M

16 layers, each with one shared expert and 64 small routed experts, of which every token uses 4. Routing is balanced without an auxiliary loss (Quantile Balancing). A new 49,152-token tokenizer, trained for this model, keeps code indentation intact and covers ten languages.

Every design switch was A/B tested at 200M tokens before the full run, and the losers are published too. Muon beat AdamW clearly: 3.32 vs 4.06 loss, against seed noise of 0.03.

Data: 60% permissively licensed code, 28% English, 12% Indonesian, Malay, Chinese, Japanese, Korean, Tamil, Hindi and Urdu, each document checked with my own language-ID model. HumanEval and MBPP were filtered out of the training data.

Bits per byte at the end of pretraining

Lower is better, comparable across tokenizers. Held-out text per language, and held-out code.

02 First model · quipu-114m

A 114M-parameter decoder, sized for one laptop.

Every number here was measured on the machine that trained it, an RTX 5060 Laptop GPU with 8 GB.

114,114,048parameters
3Btraining tokens
18.4ktokens / second, bf16
45 h 22 mtraining time, zero restarts

Architecture

Decoder-only transformer in the modern style: RMSNorm, rotary position embeddings, SwiGLU feed-forward, and grouped-query attention with 12 query heads sharing 4 key/value heads. Input and output embeddings are tied, and the context is 1,024 tokens.

Short context by design. Long-range recall is meant to come from memory rather than a longer attention window.

Token embedding tied with output
RMSNorm → Attention GQA 12q / 4kv · RoPE
RMSNorm → Feed-forward SwiGLU
× N blocks
Final RMSNorm
Output head shares embedding weights

03 Results · quipu-114m

45 hours of training: what it learned.

All 5,722 steps ran unattended over a weekend: no restarts, no skipped steps, no data repeated. Validation loss on held-out FineWeb-Edu text, measured every 100 steps:

fig. 02 · text validation lossstep
6.38 → 3.27 text validation loss. Most of the gain comes in the first 1,000 steps (0.5B tokens); after that it is slow and steady, and nearly flat by the last few hundred steps.

Watch it learn

Same prompt, greedy decoding, at each saved milestone.

verdict

It sounds right but often isn't.

The final model says the printing press was invented "in 1848 by the French printer Pierre-Louis Leclerc". That is normal for a 114M base model: it writes fluent sentences, but the facts are often wrong, and it still repeats itself under greedy decoding.

caveat · code

The code loss of 0.97 flatters it.

GPT-2's tokenizer splits indentation into single spaces: 36.5% of the code validation tokens are pure whitespace, against 2.0% for text. They are easy to predict, and greedy code output collapses into runs of spaces. That measurement is why the next model gets a code-aware tokenizer.

NextDistillation