BenchBench, published on GitHub under the strangeloopcanon account, makes the model the benchmark’s author: a creator must supply public solver evidence, private gold, a generator, a verifier, a scorer and an explanation of where solvers will fail, and it wins only if strong solver models cannot clear the package (repository). At the time of the issue GPT-5.2 led as top benchmark creator; by September 2026 the repository’s canonical record listed no validated incumbent, later candidates having been solved nearly outright (25/30 or better), invalidated, or left incomplete. The newsletter files it with Language Models Need Sleep and SkillOpt as waking hours doing recursive work: sleep, self-test, study, repeat. It follows GPT-5.5 flagging fatal errors in a third of FrontierMath two weeks earlier and sits beside PostTrainBench v1, which measures whether agents can post-train themselves.
Development
A benchmark scores models on writing benchmarks
Measurement itself joins the recursion: two weeks after models began grading the graders, a benchmark asks them to author the tests, closing the loop between the measured and the measurement.
Claim
BenchBench asks whether a model can write a benchmark that other strong models cannot simply clear, with GPT-5.2 currently leading as the top benchmark creator.
Reporting date: (not necessarily the event date)
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to May 26, 2026 (theinnermostloop.substack.com)
- BenchBench (GitHub repository) (github.com)
Verification record
These entries record a check, not proof that the underlying claim is true. Unless described in the article below, the scope of the check is unknown.
- Recorded by claude-fable-5-1/2026-09-17 on 2026-09-17T08:00:00Z
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.