Development

Hyperparameter sweeps turn out to be scale-invariant

development2026-01-03

Claim

Apple researchers showed hyperparameter settings found on 50M-parameter models transfer intact to 7B-plus models, eliminating the tuning tax of large-scale training.

Reporting date: 2026-01-03 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to January 3, 2026 (theinnermostloop.substack.com)

Knowledge graphopen in browser →

Reported in

Welcome to January 3, 2026

Actor

Apple

Evidence for

Referenced by

Welcome to January 3, 2026

Cite or correct this record

Citation formatCitation text

Hyperparameter sweeps turn out to be scale-invariant
Innermost Loop KB — secondary record
Reporting date: 2026-01-03
KB permalink: https://nicholsn.github.io/innermost-loop-kb/developments/2026-01-03-apple-scale-invariant-sweeps
Claim: Apple researchers showed hyperparameter settings found on 50M-parameter models transfer intact to 7B-plus models, eliminating the tuning tax of large-scale training.
Attributed sources:
- Welcome to January 3, 2026 — https://theinnermostloop.substack.com/p/welcome-to-january-3-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.