Development

A lab audits the benchmark its rival dominates and finds 30% broken

development2026-07-10

Claim

With Claude still ruling spatial reasoning, OpenAI audited SWE-Bench Pro — the benchmark Claude dominates — found 30% of it broken and retracted its endorsement, while Sol topped DeepSWE at 38% of Fable's cost and set a coding index record on half the tokens.

Reported metric: 30% broken

Reporting date: 2026-07-10 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to July 10, 2026 (theinnermostloop.substack.com)

Knowledge graphopen in browser →

Reported in

Welcome to July 10, 2026

Actor

Evidence for

Supersedes

A training run accidentally ingests a partner's benchmark tasks

Referenced by

Cite or correct this record

Citation formatCitation text

A lab audits the benchmark its rival dominates and finds 30% broken
Innermost Loop KB — secondary record
Reporting date: 2026-07-10
KB permalink: https://nicholsn.github.io/innermost-loop-kb/developments/2026-07-10-a-lab-audits-the-benchmark-its-rival-dominates
Claim: With Claude still ruling spatial reasoning, OpenAI audited SWE-Bench Pro — the benchmark Claude dominates — found 30% of it broken and retracted its endorsement, while Sol topped DeepSWE at 38% of Fable's cost and set a coding index record on half the tokens.
Attributed sources:
- Welcome to July 10, 2026 — https://theinnermostloop.substack.com/p/welcome-to-july-10-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.