HumanEval history

Task and data. HumanEval is a set of 164 hand-written Python function-synthesis tasks defined by a signature, docstring, body placeholder, and hidden unit tests.

Scoring. The paper reports pass@k, an estimate of the probability that at least one of k generated samples passes all tests; higher is better for a fixed k and sampling setup.

What it measures. small, self-contained Python code generation from natural-language function specifications under functional tests.

Comparison rules. Never compare pass@1 to pass@10 or pass@100 as one metric. HumanEval+, language ports, and modified test suites are separate lineages.

Access and licensing. The Compendium cites aggregate results only and does not reproduce prompts, reference code, or tests.

Task and data. HumanEval is a set of 164 hand-written Python function-synthesis tasks defined by a signature, docstring, body placeholder, and hidden unit tests.

Scoring. The paper reports pass@k, an estimate of the probability that at least one of k generated samples passes all tests; higher is better for a fixed k and sampling setup.

What it measures. small, self-contained Python code generation from natural-language function specifications under functional tests.

Comparison rules. Never compare pass@1 to pass@10 or pass@100 as one metric. HumanEval+, language ports, and modified test suites are separate lineages.

Access and licensing. The Compendium cites aggregate results only and does not reproduce prompts, reference code, or tests.

Task and data. HumanEval is a set of 164 hand-written Python function-synthesis tasks defined by a signature, docstring, body placeholder, and hidden unit tests.

Scoring. The paper reports pass@k, an estimate of the probability that at least one of k generated samples passes all tests; higher is better for a fixed k and sampling setup.

What it measures. small, self-contained Python code generation from natural-language function specifications under functional tests.

What it does not measure. repository-scale software engineering, maintainability, security, test completeness, multi-file work, tool use, or correctness beyond the supplied tests.

Limitations. The benchmark is small and public, unit tests are not exhaustive, pass@k depends on sampling temperature and candidate count, and execution requires strong sandboxing.

Comparison rules. Never compare pass@1 to pass@10 or pass@100 as one metric. HumanEval+, language ports, and modified test suites are separate lineages.

Access and licensing. The Compendium cites aggregate results only and does not reproduce prompts, reference code, or tests.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.