Theme

Benchmarks retire faster than they can be built

Explore this theme’s evidence over time →

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to December 26, 2025 (theinnermostloop.substack.com)

Tests stop bounding the thing they measure and are abandoned by their own authors. The replacement is not a harder exam but open mathematics — measurement conceding to research.

Development timeline

Newest reporting dates first. Reporting dates may differ from event dates.

2026-09-17: Astra leads rebuilt MathArena tests

2026-08-23: An agent architecture lifts a model from 30% to a perfect 100

2026-08-15: A lab reports its AI R&D evaluations have saturated

2026-08-13: Stock tooling turns 30% into 96.2% for about $540

2026-08-12: A model reaches the human baseline on a benchmark built to be impossible

2026-08-04: A benchmark tests rebuilding whole projects from scratch and passing every test

2026-07-26: A model quadruples the best score on the hardest reasoning benchmark

2026-07-17: AI forecasters become statistically indistinguishable from superforecasters

2026-07-17: A harness hits 99% on a benchmark with zero weight changes

2026-07-10: A model beats an ARC-AGI-3 game and hits 92.5% on ARC-AGI-2

2026-07-09: A model completes over half of real workflow objectives without breaking rules

2026-07-03: Agent learning speed is found to double every three months

2026-07-02: Agents graded on taste solve under a quarter of senior-level tasks

2026-06-22: Frontier models flail at long-horizon strategy in a video game

2026-06-21: A biology model raises the pass rate on 750 expert tasks

2026-06-13: An AI audit finds errors in 42% of a benchmark's problems

2026-06-10: One model tops effectively every board at once

2026-06-08: A benchmark finds long-horizon success turns on stubbornness

2026-06-07: Frontier agents resolve under a fifth of a multi-hour task marathon

2026-06-02: The best model reaches 1.5% of human efficiency on ARC-AGI-3

2026-05-28: A software benchmark ships contamination-free repos and behavioral verifiers

2026-05-26: A benchmark scores models on writing benchmarks

2026-05-25: A benchmark spans twenty-three real deployable SaaS systems

2026-05-22: A model matches superforecaster performance

2026-05-17: A competitive security format breaks as scores track token budgets

2026-05-16: Rank by exploitation and one model leads; rank by economic value and it reverses

2026-05-15: A meta-system builds its own harnesses and sets a coding record

2026-05-15: Agents given idle compute beat the human speedrun baseline

2026-05-15: A model writes a console emulator from scratch half the time

2026-05-14: A model becomes the first to clear both national cyber ranges

2026-05-13: A meta-eval maps twelve benchmarks onto an implied IQ of 136

2026-05-12: A model flags fatal errors in a third of a benchmark's problems

2026-05-09: The autonomy horizon exceeds what the measuring suite can gauge

2026-05-08: A benchmark asks agents to rebuild whole codebases from a binary

2026-05-06: Hallucinated claims fall 52.5% on high-stakes prompts

2026-05-05: A model closes on the human baseline for reading rooms from photos

2026-05-04: A US body puts Chinese models eight months behind

2026-05-03: The hardest benchmark starts to move

2026-04-29: A fresh olympiad benchmark more than doubles in a single release

2026-04-26: One release takes math, search, economic value and software engineering

2026-04-16: A model becomes the first to fully crack a 32-step network attack

2026-04-08: A benchmark measures capability and safety in the same environment

2026-04-08: One model sweeps the board and jumps the capability curve

2026-03-31: Agent control logic becomes a portable natural-language object

2026-03-29: A mathematics olympiad goes from 5% to 95% in twelve months

2026-03-28: Problems are removed from a benchmark for being insufficiently notable

2026-03-28: A benchmark measures whether models can change each other's minds

2026-03-27: A new benchmark returns frontier models to near zero

2026-03-20: Autoformalization gets benchmarked against qualifying exams

2026-03-18: DeepMind decomposes general intelligence into ten faculties

2026-03-12: A benchmark measures whether agents can post-train themselves

2026-03-09: A metric measures how willingly models help fabricate papers

2026-03-09: Musk expects full AGI by year end as Brockman waves off benchmarks

2026-03-08: A physics benchmark goes from 9% to 30% in four months

2026-03-06: A model matches human professionals 83% of the time

2026-03-05: A benchmark rewards refusing to answer nonsense

2026-02-26: Karpathy says it is hard to convey how much programming changed in two months

2026-02-24: Two hours of video fits in a million-token window

2026-02-23: A model solves a FrontierMath problem no model had solved

2026-02-20: Gemini 3.1 Pro takes the index lead at half the evaluation cost

2026-02-14: A distinction between measurement that records and measurement that creates

2026-02-13: Seven people on Earth can still beat it at competitive programming

2026-02-12: An internal model takes 91.9% on advanced proof benchmarks

2026-02-11: Orchestrating three labs' models takes 55% on Humanity's Last Exam

2026-02-08: Noam Brown predicts autonomy horizons will become unmeasurable

2026-02-08: A model takes first on a research physics benchmark

2026-02-07: Mathematicians publish problems with encrypted solutions

2026-02-07: Opus 4.6 takes the top index spot as markets price the next month

2026-02-06: A lab says it has built the last open-answer benchmark it can

2026-02-06: Opus 4.6 sets a new high on Humanity's Last Exam

2026-02-06: A benchmark record is beaten under thirty minutes later

2026-02-05: An ensemble of three labs' models hits 94.5% on ARC-AGI

2026-02-02: A model beats Pokémon Emerald while thinking 71% of the time

2026-01-30: A consciousness model ranks 2024 models below chickens

2026-01-26: A model beats the S&P with real money

2026-01-26: A dataset curates a thousand open problems

2026-01-25: Anthropic opens its exam and asks for humans who can beat the model

2026-01-25: FrontierMath Tier 4 jumps from 19% to 31%

2026-01-25: A model tops a new legal reasoning benchmark at 52%

2026-01-14: Harmonic aims $295M at the Millennium problems

2026-01-13: A benchmark gives models $10,000 to bet

2026-01-12: Erdős #397 falls, the third this week

2026-01-10: A prover solves all twelve Putnam problems in Lean

2026-01-07: Frontier models play live chess against each other

2026-01-06: SkillsBench audits agents more strictly

2026-01-02: Robotaxi expansion gets a leaderboard

2025-12-31: GPT-5.2 Pro takes 29.2% on FrontierMath Tier 4

2025-12-26: ARC Prize declares its own benchmarks saturated

Knowledge graphopen in browser →

Cite or correct this record

Citation formatCitation text

Benchmarks retire faster than they can be built
Innermost Loop KB — secondary record
Reporting date: unknown
KB permalink: https://nicholsn.github.io/innermost-loop-kb/themes/benchmark-saturation
Attributed sources:
- Welcome to December 26, 2025 — https://theinnermostloop.substack.com/p/welcome-to-december-26-2025
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.