Under two minutes, and the scaling ratio everyone trained on turns out not to bind. The 99.3-second run on the NanoGPT speedrun used a bigram hash embedding, a hashed lookup table keyed on token pairs, and reached the leaderboard’s fixed FineWeb validation-loss target after seeing fewer training tokens than the model has parameters, inverting the Chinchilla token-to-parameter ratio (X post). It takes 7.6 seconds off the 106.9-second record of twelve days earlier, which had come from compiler kernel hacking rather than architecture, and it is the first sub-100-second entry in the corpus’s speedrun series; the record next falls to 88.1 seconds in late February. Like DeepSeek’s U-shaped memory law nine days earlier, it trades learned computation for static lookup.
Development
The speedrun breaks 100 seconds with fewer tokens than parameters
The newsletter files the record under scaling laws being rewritten: the sub-100-second mark matters less than the fact that it was reached by training on fewer tokens than the model has parameters, treating the Chinchilla ratio as a habit rather than a bound.
Claim
A NanoGPT speedrun record of 99.3 seconds was set with a bigram hash embedding, using fewer training tokens than parameters in a departure from Chinchilla ratios.
Reported metric: 99.3 s
Reporting date: (not necessarily the event date)
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to January 24, 2026 (theinnermostloop.substack.com)
- NanoGPT speedrun record of 99.3 seconds with a bigram hash embedding (X post) (x.com)
Verification record
These entries record a check, not proof that the underlying claim is true. Unless described in the article below, the scope of the check is unknown.
- Recorded by claude-fable-5-1/2026-09-17 on 2026-09-17T08:00:00Z
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.