MATH
MATH evaluates competition-style mathematical problem solving across algebra, geometry, number theory, counting and probability, and related school-level areas.
MATH MATH 2021 paper release
Permanent entry- Task and data
- MATH contains 12,500 competition-mathematics problems across seven subjects and five difficulty levels, each paired with a worked solution in the released dataset.
- Primary metric
- The paper primarily reports exact-answer accuracy after normalization of equivalent final-answer formats; higher is better within the same split and generation rules.
- What it measures
- competition-style mathematical problem solving across algebra, geometry, number theory, counting and probability, and related school-level areas.
- Comparison rules
- Separate few-shot GPT-3 from AMPS-pretrained and fine-tuned GPT-2 results. Modern MATH variants and scoring pipelines are different lineages.
- Access and licensing
- The Compendium does not reproduce problems, figures, or solutions. Code licensing does not automatically grant rights to every source problem.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-2 1.5Bmath-2021:test:amps-pretrained:fine-tuned | 6.9%higher, percent | exact-answer accuracyhistorical paper baseline | MATH test setFine-tuned model pretrained on the paper's AMPS auxiliary corpus; aggregate across seven subjects.This setup includes AMPS pretraining and is not directly comparable to the paper's GPT-3 API rows. | MATH authorsPublished 2021-03-05, reviewed 2026-07-13original-paper, published-third-party |
| GPT-3 175Bmath-2021:test:gpt3:few-shot | 5.2%higher, percent | exact-answer accuracyhistorical paper baseline | MATH test setFew-shot GPT-3 API evaluation; no AMPS pretraining; the exact API snapshot is not stated beyond the 175B model label.Few-shot and fine-tuned rows use different adaptation regimes. | MATH authorsPublished 2021-03-05, reviewed 2026-07-13original-paper, published-third-party |
| Three-time IMO gold medalistmath-2021:human:n20:one-hour | 90%higher, percent | accuracysmall-sample human anchor | 20 randomly sampled MATH test problemsOne-hour, by-hand human study; 18 of 20 correct.A single expert on 20 items is illustrative, not a precise benchmark-wide human ceiling. | MATH authorsPublished 2021-03-05, reviewed 2026-07-13human-baseline, published-third-party |
Claims and evidence
Claim ledger, 2MATH contains 12,500 competition-mathematics problems across seven subjects and five difficulty levels, each paired with a worked solution in the released dataset.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.