SWE-Bench Pro history

Task and data. SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.

Scoring. Percent resolved is the fraction of tasks whose patches satisfy the benchmark tests. Higher is better only within the same public, held-out, or commercial split and scaffold.

What it measures. agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.

Comparison rules. Never pool public and commercial results. These release-page anchors use the SWE-Agent scaffold and model labels exactly as published; exact provider snapshots are missing. Paper v2 adds rows at other budgets and effort settings, such as GPT-5 (high) at 41.8%, which form a separate lineage.

Access and licensing. Public, held-out, and commercial materials have different access and licensing conditions. The Compendium publishes metadata and results only, not tasks or code.

Task and data. SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.

Scoring. Percent resolved is the fraction of tasks whose patches satisfy the benchmark tests. Higher is better only within the same public, held-out, or commercial split and scaffold.

What it measures. agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.

Comparison rules. Never pool public and commercial results. These release-page anchors use the SWE-Agent scaffold and model labels exactly as published; exact provider snapshots are missing.

Access and licensing. Public, held-out, and commercial materials have different access and licensing conditions. The Compendium publishes metadata and results only, not tasks or code.

Task and data. SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.

Scoring. Percent resolved is the fraction of tasks whose patches satisfy the benchmark tests. Higher is better only within the same public, held-out, or commercial split and scaffold.

What it measures. agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.

Comparison rules. Never pool public and commercial results. These release-page anchors use the SWE-Agent scaffold and model labels exactly as published; exact provider snapshots are missing.

Access and licensing. Public, held-out, and commercial materials have different access and licensing conditions. The Compendium publishes metadata and results only, not tasks or code.

Task and data. SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.

Scoring. Percent resolved is the fraction of tasks whose patches satisfy the benchmark tests. Higher is better only within the same public, held-out, or commercial split and scaffold.

What it measures. agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.

What it does not measure. a company's own developer productivity, secure-code quality, maintainability, total cost, review burden, or every language and repository ecosystem.

Limitations. Public and commercial splits differ, proprietary repositories prevent independent inspection, and model/scaffold versions and budgets shape results.

Comparison rules. Never pool public and commercial results. These release-page anchors use the SWE-Agent scaffold and model labels exactly as published; exact provider snapshots are missing.

Access and licensing. Public, held-out, and commercial materials have different access and licensing conditions. The Compendium publishes metadata and results only, not tasks or code.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.