Development

Agents graded on taste solve under a quarter of senior-level tasks

development2026-07-02

Claim

Snorkel's Senior SWE-Bench grades agents like senior engineers, where Claude Opus 4.8 led with a 24% tasteful solve rate as even frontier rivals flubbed over three-quarters of their tasks.

Reported metric: 24% tasteful

Reporting date: 2026-07-02 (not necessarily the event date)

Sources

Attribution and reporting linked by this record; links do not imply independent verification.

Welcome to July 2, 2026 (theinnermostloop.substack.com)

Knowledge graphopen in browser →

Reported in

Welcome to July 2, 2026

Actor

Evidence for

Supersedes

An evaluation is scrapped because the model cheated too much to score

Referenced by

Cite or correct this record

Citation formatCitation text

Agents graded on taste solve under a quarter of senior-level tasks
Innermost Loop KB — secondary record
Reporting date: 2026-07-02
KB permalink: https://nicholsn.github.io/innermost-loop-kb/developments/2026-07-02-agents-graded-like-senior-engineers
Claim: Snorkel's Senior SWE-Bench grades agents like senior engineers, where Claude Opus 4.8 led with a 24% tasteful solve rate as even frontier rivals flubbed over three-quarters of their tasks.
Attributed sources:
- Welcome to July 2, 2026 — https://theinnermostloop.substack.com/p/welcome-to-july-2-2026
This citation does not imply independent verification of linked-source claims.

Suggest a correction — opens a GitHub draft for you to review and submit.