Averaging token embeddings gives a threefold pretraining speedup

Claim

Nous Research's token superposition training delivers a two- to threefold wall-clock pretraining speedup at matched compute by averaging contiguous bags of token embeddings, with no architecture change required.

Reported metric: 2-3x

Reporting date: 2026-05-14 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

• Welcome to May 14, 2026 (theinnermostloop.substack.com)