Development

A model flags fatal errors in a third of a benchmark's problems

development2026-05-12

Claim

OpenAI's Noam Brown revealed that GPT-5.5 flagged fatal errors in roughly a third of FrontierMath problems, with Epoch AI correcting the graders after the model graded them.

Reported metric: ~1/3 of problems

Reporting date: 2026-05-12 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to May 12, 2026 (theinnermostloop.substack.com)

March removed problems for being unworthy. May finds them wrong.

Knowledge graphopen in browser →

Reported in

Welcome to May 12, 2026

Actor

Evidence for

Supersedes

Problems are removed from a benchmark for being insufficiently notable

Referenced by

Cite or correct this record

Citation formatCitation text

A model flags fatal errors in a third of a benchmark's problems
Innermost Loop KB — secondary record
Reporting date: 2026-05-12
KB permalink: https://nicholsn.github.io/innermost-loop-kb/developments/2026-05-12-the-model-grades-the-graders
Claim: OpenAI's Noam Brown revealed that GPT-5.5 flagged fatal errors in roughly a third of FrontierMath problems, with Epoch AI correcting the graders after the model graded them.
Attributed sources:
- Welcome to May 12, 2026 — https://theinnermostloop.substack.com/p/welcome-to-may-12-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.