SWE-Bench Pro
SWE-Bench Pro evaluates agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.
SWE-Bench Pro SWE-Bench Pro September 2025 release
Permanent entry- Task and data
- SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.
- Primary metric
- Percent resolved is the fraction of tasks whose patches satisfy the benchmark tests. Higher is better only within the same public, held-out, or commercial split and scaffold.
- What it measures
- agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.
- Comparison rules
- Never pool public and commercial results. These release-page anchors use the SWE-Agent scaffold and model labels exactly as published; exact provider snapshots are missing.
- Access and licensing
- Public, held-out, and commercial materials have different access and licensing conditions. The Compendium publishes metadata and results only, not tasks or code.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| OpenAI GPT-5 (exact snapshot not stated by source)swebench-pro-2025:public:swe-agent | 23.3%higher, percent | resolvedofficial release result | public datasetSWE-Agent scaffold; the release page does not name the exact GPT-5 snapshot or restate every inference parameter.Do not infer a provider snapshot or compare this with commercial-split results. | Scale AIPublished 2025-09-19, reviewed 2026-07-13original-paper, published-third-party |
| Claude Opus 4.1 (exact snapshot not stated by source)swebench-pro-2025:public:swe-agent | 23.1%higher, percent | resolvedofficial release result | public datasetSWE-Agent scaffold; the release page does not name the exact Claude Opus 4.1 snapshot or restate every inference parameter.A 0.2-point difference should not be treated as decisive without uncertainty and repeat-run detail. | Scale AIPublished 2025-09-19, reviewed 2026-07-13original-paper, published-third-party |
| Claude Opus 4.1 (exact snapshot not stated by source)swebench-pro-2025:commercial:swe-agent | 17.8%higher, percent | resolvedofficial private-split result | private commercial subsetSWE-Agent scaffold; proprietary repository evaluation. Exact model snapshot and full task artifacts are unavailable on the release page.Private tasks cannot be independently inspected and are a different comparison class from public tasks. | Scale AIPublished 2025-09-19, reviewed 2026-07-13original-paper, published-third-party |
| OpenAI GPT-5 (exact snapshot not stated by source)swebench-pro-2025:commercial:swe-agent | 14.9%higher, percent | resolvedofficial private-split result | private commercial subsetSWE-Agent scaffold; proprietary repository evaluation. Exact model snapshot and full task artifacts are unavailable on the release page.Private tasks cannot be independently inspected and are a different comparison class from public tasks. | Scale AIPublished 2025-09-19, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.