01 Distillation

Keep only the answers that pass their tests.

The chat model could barely write code: greedy decoding solved 5 of MBPP's 257 problems. So an open-weight teacher, Qwen2.5-Coder-1.5B-Instruct (Apache-2.0), answered 5,000 coding problems, and every answer was run against the problem's unit tests in a sandbox. Only the 3,277 that passed were kept. The chat fine-tune was then repeated with those added.

1.9% → 7.4%MBPP pass@1, greedy (5 → 19 of 257)
0% → 0.6%HumanEval pass@1, greedy (0 → 1 of 164)
for comparison

About 4x better, still far behind bigger models.

SantaCoder (1.1B) reports around 35% on MBPP and Qwen2.5-Coder-1.5B about 69%, with different settings. Quipu uses 149M parameters per token and saw 4.26B tokens. It is not competing with them yet.

done · v2

The same idea, at a much bigger scale.

A 7B teacher, Qwen2.5-Coder-7B-Instruct (Apache-2.0), generated 638,342 answers, and every one was run against its unit tests in a sandbox. 498,813 passed and were kept: 78.1%.

Generation ran on a rented RTX 3090, the laptop and Modal H100s; verification ran on Modal CPUs.

638,342teacher answers generated (v2)
498,813passed their unit tests and kept
78.1%pass rate

The demo runs on a small CPU machine that sleeps when idle: the first visit can take about 15 seconds to wake it.

NextThe data