CoBench is Anthropic’s internal evaluation, defined in section 3.4.3 of its redacted August 2026 Risk Report: a model is placed at a historical point in Anthropic’s infrastructure, given a snapshot of the codebase, logs, messaging and docs, and asked to diagnose the root causes of issues its engineers actually solved, across 449 problems drawn from February to April 2026 (risk report). The 85% threshold is the lab’s own estimate of the score a model truly capable of fully substituting for its research staff would reach; watchers applied it to the redacted gap between Model 2 and Mythos 5, 12.5 points, to conclude that “2027 is the takeoff” (X). It stands beside PostTrainBench as one of the few named instruments for AI doing AI research, introduced at the moment the lab says its task-based AI R&D evaluations have saturated.
Benchmark
CoBench v2
Anthropic's internal AI R&D evaluation, defined in its August 2026 Risk Report, whose 85% threshold the lab itself estimates would mark full researcher substitution and on which Model 2 beat Mythos 5 by 12.5 points.
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to August 15, 2026 (theinnermostloop.substack.com)
Development timeline
Newest reporting dates first. Reporting dates may differ from event dates.
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.