Development

A benchmark asks agents to rebuild whole codebases from a binary

development2026-05-08

Claim

The ProgramBench benchmark asks agents to rebuild full codebases from a binary alone, where Opus 4.7 leads at 3% almost resolved and zero percent fully solved.

Reported metric: 3% / 0%

Reporting date: 2026-05-08 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to May 8, 2026 (theinnermostloop.substack.com)

A humbling reminder that the ladder still has rungs above.

Knowledge graphopen in browser →

Reported in

Welcome to May 8, 2026

Actor

Anthropic

Evidence for

Supersedes

The hardest benchmark starts to move

Referenced by

Cite or correct this record

Citation formatCitation text

A benchmark asks agents to rebuild whole codebases from a binary
Innermost Loop KB — secondary record
Reporting date: 2026-05-08
KB permalink: https://nicholsn.github.io/innermost-loop-kb/developments/2026-05-08-rebuilding-a-codebase-from-a-binary
Claim: The ProgramBench benchmark asks agents to rebuild full codebases from a binary alone, where Opus 4.7 leads at 3% almost resolved and zero percent fully solved.
Attributed sources:
- Welcome to May 8, 2026 — https://theinnermostloop.substack.com/p/welcome-to-may-8-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.