HumanEval
HumanEval evaluates small, self-contained Python code generation from natural-language function specifications under functional tests.
HumanEval HumanEval paper v2; 164 Python tasks
Permanent entry- Task and data
- HumanEval is a set of 164 hand-written Python function-synthesis tasks defined by a signature, docstring, body placeholder, and hidden unit tests.
- Primary metric
- The paper reports pass@k, an estimate of the probability that at least one of k generated samples passes all tests; higher is better for a fixed k and sampling setup.
- What it measures
- small, self-contained Python code generation from natural-language function specifications under functional tests.
- Comparison rules
- Never compare pass@1 to pass@10 or pass@100 as one metric. HumanEval+, language ports, and modified test suites are separate lineages.
- Access and licensing
- The Compendium cites aggregate results only and does not reproduce prompts, reference code, or tests.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-J 6Bhumaneval-v2:test:pass@1 | 11.62%higher, percent | pass@1historical paper result | 164 HumanEval taskspass@1 estimated from 200 sampled programs per task under the paper's sampling and execution protocol.A pass means supplied tests passed; it is not proof of complete program correctness. | OpenAI Codex paper authorsPublished 2021-07-14, reviewed 2026-07-13original-paper, published-third-party |
| Codex-12Bhumaneval-v2:test:pass@1 | 28.81%higher, percent | pass@1historical paper result | 164 HumanEval taskspass@1 estimated from 200 sampled programs per task under the paper's Codex evaluation protocol.The early research model label is not interchangeable with later products named Codex. | OpenAI Codex paper authorsPublished 2021-07-14, reviewed 2026-07-13original-paper, published-third-party |
| Codex-S-12Bhumaneval-v2:test:pass@1:supervised | 37.7%higher, percent | pass@1historical paper result | 164 HumanEval taskspass@1 estimated from 200 sampled programs per task after supervised fine-tuning on standalone functions.The supervised adaptation changes the comparison class relative to base Codex. | OpenAI Codex paper authorsPublished 2021-07-14, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2HumanEval is a set of 164 hand-written Python function-synthesis tasks defined by a signature, docstring, body placeholder, and hidden unit tests.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.