SWE-bench Verified — historical release
SWE-bench Verified — historical release evaluates agent-plus-model ability to navigate Python repositories and produce test-passing patches for selected real-world issues.
SWE-bench Verified — historical release 500-instance 2024 release; historical OpenAI snapshot
Permanent entry- Task and data
- SWE-bench Verified is a human-filtered set of 500 real GitHub issue-resolution tasks whose problem statements, tests, and environments were reviewed for feasibility.
- Primary metric
- Percent resolved is the share of instances whose generated patch applies and passes both FAIL_TO_PASS and PASS_TO_PASS tests; higher is better for a fixed scaffold and harness.
- What it measures
- agent-plus-model ability to navigate Python repositories and produce test-passing patches for selected real-world issues.
- Comparison rules
- Treat the 2024 OpenAI figure as historical. Current mini-SWE-agent releases 1.x and 2.x are explicitly not necessarily comparable, and arbitrary-scaffold results form other lineages.
- Access and licensing
- Repository issues, code, and tests retain their original project licenses. The Compendium does not reproduce issue text, patches, tests, or repository assets.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| gpt-4o-2024-05-13 with Agentlessswebench-verified-2024:openai-study:agentless:single-seed | 33.2%higher, percent | resolvedhistorical; not a current leaderboard rank | 500-instance SWE-bench VerifiedBest-performing open-source scaffold in OpenAI's study; one seed using closest-documented or default hyperparameters.OpenAI states the run may differ from official leaderboard reports; later harness and scaffold results are separate. | OpenAI and SWE-benchPublished 2024-08-13, reviewed 2026-07-13historical-snapshot, published-third-party |
Claims and evidence
Claim ledger, 2SWE-bench Verified is a human-filtered set of 500 real GitHub issue-resolution tasks whose problem statements, tests, and environments were reviewed for feasibility.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.