SWE-bench Verified — historical release history

Task and data. SWE-bench Verified is a human-filtered set of 500 real GitHub issue-resolution tasks whose problem statements, tests, and environments were reviewed for feasibility.

Scoring. Percent resolved is the share of instances whose generated patch applies and passes both FAIL_TO_PASS and PASS_TO_PASS tests; higher is better for a fixed scaffold and harness.

What it measures. agent-plus-model ability to navigate Python repositories and produce test-passing patches for selected real-world issues.

Comparison rules. Treat the 2024 OpenAI figure as historical. Current mini-SWE-agent releases 1.x and 2.x are explicitly not necessarily comparable, and arbitrary-scaffold results form other lineages.

Access and licensing. Repository issues, code, and tests retain their original project licenses. The Compendium does not reproduce issue text, patches, tests, or repository assets.

Task and data. SWE-bench Verified is a human-filtered set of 500 real GitHub issue-resolution tasks whose problem statements, tests, and environments were reviewed for feasibility.

Scoring. Percent resolved is the share of instances whose generated patch applies and passes both FAIL_TO_PASS and PASS_TO_PASS tests; higher is better for a fixed scaffold and harness.

What it measures. agent-plus-model ability to navigate Python repositories and produce test-passing patches for selected real-world issues.

Comparison rules. Treat the 2024 OpenAI figure as historical. Current mini-SWE-agent releases 1.x and 2.x are explicitly not necessarily comparable, and arbitrary-scaffold results form other lineages.

Access and licensing. Repository issues, code, and tests retain their original project licenses. The Compendium does not reproduce issue text, patches, tests, or repository assets.

Task and data. SWE-bench Verified is a human-filtered set of 500 real GitHub issue-resolution tasks whose problem statements, tests, and environments were reviewed for feasibility.

Scoring. Percent resolved is the share of instances whose generated patch applies and passes both FAIL_TO_PASS and PASS_TO_PASS tests; higher is better for a fixed scaffold and harness.

What it measures. agent-plus-model ability to navigate Python repositories and produce test-passing patches for selected real-world issues.

What it does not measure. general software-engineering productivity, code quality beyond tests, security, novel private-code generalization, cost, or model-only ability independent of scaffolding.

Limitations. Tasks come from public repositories and can be contaminated; scores vary materially with agent scaffold, harness release, tools, retries, and compute budget.

Comparison rules. Treat the 2024 OpenAI figure as historical. Current mini-SWE-agent releases 1.x and 2.x are explicitly not necessarily comparable, and arbitrary-scaffold results form other lineages.

Access and licensing. Repository issues, code, and tests retain their original project licenses. The Compendium does not reproduce issue text, patches, tests, or repository assets.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.