Development

One module proposes embeddings and another solves for the fixed point

development2026-05-15

Claim

New attractor models let one module propose embeddings while another solves for the fixed point, taming looped transformers enough for a 770-million parameter model to outrun a 1.3-billion one trained on twice the tokens.

Reported metric: 770M beats 1.3B

Reporting date: 2026-05-15 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to May 15, 2026 (theinnermostloop.substack.com)

Knowledge graphopen in browser →

Reported in

Welcome to May 15, 2026

Evidence for

Supersedes

Averaging token embeddings gives a threefold pretraining speedup

Referenced by

Cite or correct this record

Citation formatCitation text

One module proposes embeddings and another solves for the fixed point
Innermost Loop KB — secondary record
Reporting date: 2026-05-15
KB permalink: https://nicholsn.github.io/innermost-loop-kb/developments/2026-05-15-attractor-models-tame-looped-transformers
Claim: New attractor models let one module propose embeddings while another solves for the fixed point, taming looped transformers enough for a 770-million parameter model to outrun a 1.3-billion one trained on twice the tokens.
Attributed sources:
- Welcome to May 15, 2026 — https://theinnermostloop.substack.com/p/welcome-to-may-15-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.