The eighth link. Engineers joke the series has to stop at zero. Larry Dial’s record log attributes the 3.5-second gain to Paired Head Attention (pull request #191), which lets queries attend to keys in the neighbouring head so each query retrieves two values per target position, worth about 3% for zero added parameters (X post). Where the 113.7-second mark came from optimizer and parameter-layout tuning, this record on the NanoGPT speedrun comes from the attention architecture, which is why it also sits under architecture-of-mind. The next two records take different routes again: compiler kernel hacking four days later and a bigram hash embedding that breaks 100 seconds.
Development
The speedrun record falls to 109.2 seconds on a new attention mechanism
An issue that opens on the marginal cost of intelligence trending to zero leads with this record, and the gain comes from the attention architecture itself rather than from systems tuning.
Claim
The NanoGPT speedrun record fell to 109.2 seconds using an attention mechanism that retrieves two values per target position instead of one.
Reported metric: 109.2 s
Reporting date: (not necessarily the event date)
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to January 8, 2026 (theinnermostloop.substack.com)
- New NanoGPT Speedrun WR at 109.2 (-3.5s) (x.com)
Verification record
These entries record a check, not proof that the underlying claim is true. Unless described in the article below, the scope of the check is unknown.
- Recorded by claude-fable-5-1/2026-09-17 on 2026-09-17T08:00:00Z
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.