Theme

The measured start correcting the measurement

Explore this theme’s evidence over time →

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to May 12, 2026 (theinnermostloop.substack.com)

Models finding errors in the problems set for them, and evaluators revising their own answer keys in response. The instrument stops being independent of the thing it measures.

Development timeline

Newest reporting dates first. Reporting dates may differ from event dates.

2026-07-16: Five adversarial reviewer personas beat human analysis in 15 of 20 comparisons

2026-07-10: A lab audits the benchmark its rival dominates and finds 30% broken

2026-07-08: A model finds edge-case bugs in a rival model's own code

2026-06-13: An AI audit finds errors in 42% of a benchmark's problems

2026-05-28: A software benchmark ships contamination-free repos and behavioral verifiers

2026-05-26: A benchmark scores models on writing benchmarks

2026-05-16: A loyalty-fraud study is withdrawn for hallucinated footnotes

2026-05-13: A meta-eval maps twelve benchmarks onto an implied IQ of 136

2026-05-12: A model flags fatal errors in a third of a benchmark's problems

Knowledge graphopen in browser →

Cite or correct this record

Citation formatCitation text

The measured start correcting the measurement
Innermost Loop KB — secondary record
Reporting date: unknown
KB permalink: https://nicholsn.github.io/innermost-loop-kb/themes/models-audit-their-benchmarks
Attributed sources:
- Welcome to May 12, 2026 — https://theinnermostloop.substack.com/p/welcome-to-may-12-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.