SWE-bench Verified is the human-validated subset of SWE-bench that OpenAI released in 2024 so that scores reflect solvable, unambiguous GitHub issues (announcement). The corpus uses it as a clock: the first issue reported Anthropic’s models improving about 2% a month toward saturation by late 2026, and Epoch AI later found 15% of the score available from prompt restructuring alone. Its harder sibling is SWE-Bench Pro, where a self-play bug-repair agent and later frontier models are scored.
Benchmark
SWE-bench Verified
OpenAI's human-validated 500-task subset of SWE-bench, the corpus's reference yardstick for agentic software engineering and the saturation clock the newsletter watches.
Sources
Attribution and reporting linked by this record; links do not imply independent verification.
- Welcome to December 11, 2025 (theinnermostloop.substack.com)
Development timeline
Newest reporting dates first. Reporting dates may differ from event dates.
Cite or correct this record
Suggest a correction — opens a GitHub draft for you to review and submit.