Simple self-distillation (SSD) samples solutions from the model at chosen temperature and truncation settings and fine-tunes on them with ordinary supervised fine-tuning; the paper (arXiv, code) lifts Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, generalizes across Qwen and Llama models at 4B, 8B and 30B in instruct and thinking variants, and traces the gain to a precision-exploration conflict in decoding that SSD resolves by suppressing distractor tails where precision matters while keeping diversity where exploration matters. Where Bilevel Autoresearch five days earlier improved a model’s research loop, this removes the loop’s scaffolding entirely and still gets a model to improve itself, the tightest form of recursive self-improvement in the corpus to date. It is followed within days by ByteDance’s in-place test-time training and UNC’s 72-hour autonomous run, and prefigures July’s finding that a student distilled from weaker teachers keeps improving.
Development
A model improves by fine-tuning on its own samples
The self-improvement loop shrinks to its minimum, with no search, no reward, no teacher and no verifier between a model and its own better samples, which the author reads as the Singularity learning to teach itself.
Claim
Apple researchers showed language models can self-improve at coding through simple self-distillation, sampling their own outputs and fine-tuning on them with no verifier, teacher or reinforcement learning, lifting one model from 42.4% to 55.3% on LiveCodeBench with gains concentrated on the hardest problems.
Reported metric: 42.4% → 55.3%
Reporting date: (not necessarily the event date)
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to April 5, 2026 (theinnermostloop.substack.com)
- Embarrassingly Simple Self-Distillation Improves Code Generation (arxiv.org)
- apple/ml-ssd: code for Simple Self-Distillation (github.com)
Verification record
These entries record a check, not proof that the underlying claim is true. Unless described in the article below, the scope of the check is unknown.
- Recorded by claude-fable-5-1/2026-09-17 on 2026-09-17T08:00:00Z
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.