01 Training code

Multi-GPU training and faster experts.

Everything so far was trained on one GPU. These two changes are for the runs after that.

verified · 14 Oct 2026

Multi-GPU data-parallel training.

Verified on real GPUs: the 2-GPU test passed. The same training code runs on one GPU or several.

mixture of experts

Expert dispatch rewritten with grouped matmuls: 3.24x faster.

Measured on a B200, in tokens per second.

before113k/s
grouped366k/s

02 Engineering

Bugs found before the long runs.

These bugs would not have shown an error. Each one was found while reviewing the code, before the run.

01 · speed

Attention was quietly using a slow kernel.

Fixing the fallback took the projected run for the original budget from ~65 hours to ~39.

02 · I/O

About 1 in 130 file renames fails transiently.

That is enough to kill a 50-hour run over a log write. Every atomic write now retries.

03 · resume

A resume could restart from zero, silently.

After days of training, with no error. It now refuses unless nothing could have been saved.

04 · numerics

One bad batch could have deleted every clean checkpoint.

Non-finite steps are now skipped, and three in a row stop the run before retention prunes the good ones.

NextDesign decisions