The paper (arXiv:2512.20605, Kobayashi, von Oswald and colleagues at Google) trains a higher-order, non-causal sequence model whose outputs steer the residual-stream activations of a base autoregressive model, compressing long activation chunks into internal controllers with learned termination conditions; reinforcing those controllers directly, which the authors call internal RL, learns from sparse rewards on grid-world and MuJoCo tasks where standard RL fine-tuning fails. The newsletter reads it as the ‘inner optimizers’ long theorized by safety researchers turning out to be remarkably effective. It sits in the machine-introspection strand after Anthropic’s Activation Oracles and DeepMind’s Gemma Scope 2: where those read a model’s internals, this one acts in them, a step toward models that rewrite their own weights in flight.
Development
Google shows inner optimizers work, via internal RL
A construct AI-safety theory had treated as a hazard, an optimizer running inside the model, is reported as a working training method: the machine's internal monologue optimizing itself.
Claim
Google researchers showed inner optimizers are remarkably effective, developing internal RL in which a higher-order model explores a base model's internal representations to learn from sparse rewards.
Reporting date: (not necessarily the event date)
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to December 27, 2025 (theinnermostloop.substack.com)
- Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning (arxiv.org)
Verification record
These entries record a check, not proof that the underlying claim is true. Unless described in the article below, the scope of the check is unknown.
- Recorded by claude-fable-5-1/2026-09-17 on 2026-09-17T08:00:00Z
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.