A 4B model dominates long context by rewriting its own weights in flight

The paper (arXiv 2604.06169) makes the final projection matrix of every MLP block the model’s fast weights, swaps test-time training’s usual reconstruction loss for an objective tied to next-token prediction, and updates in chunks so the method runs under context parallelism; applied in place to an existing 4B-parameter model it gives superior results on tasks with contexts up to 128k tokens, and the same recipe also works when pretrained from scratch. The newsletter’s framing is that ByteDance is ‘making old models smarter mid-flight’, a drop-in answer to the long-context problem that Anthropic had addressed weeks earlier with million-token windows and Prime Intellect with recursive self-calls. In the recursive-self-improvement trajectory it extends Stanford’s constant-latency test-time training from December and Apple’s teacherless self-distillation of four days earlier: the model modifies its own weights during use rather than between training runs.