01 Next

What's next.

  1. done
    Pretraining at 114M3B tokens in 45 hours on one laptop GPU. Weights are public.
  2. done
    1B mixture of experts with a code-aware tokenizer4.26B tokens, ten languages and code, $18.75 in total. The whitespace result above is why the tokenizer changed.
  3. done
    Chat fine-tune and execution-verified distillationHuman-written conversations, then teacher answers that pass their tests: MBPP 1.9% → 7.4%.
  4. done
    Distillation v2A 7B teacher, 638,342 answers, every one tested; 498,813 verified answers kept (78.1%).
  5. done
    Data v3 and multi-GPU training10B tokens from 42 open sources for $2.51; data-parallel training verified on 2 GPUs; MoE dispatch 3.24x faster.
  6. planned
    Long-context phase: 2,048 → 8,192 tokensAbout 1B tokens of training at 8k context with a larger RoPE base (~500k). The demo already stretches to 4,096 without retraining, at under 0.3% loss cost on the first 2,048 positions; this trains the model on longer text instead of only stretching it.
  7. later
    A ~1B-active modelWhen bigger compute is available.
NextContact