GPQA
GPQA evaluates graduate-level scientific question answering and the gap between domain experts, skilled outsiders, and model baselines under specified resource conditions.
GPQA 2023 paper release (Extended/Main/Diamond sets)
Permanent entry- Task and data
- GPQA Extended contains 546 expert-written multiple-choice questions in biology, physics and chemistry. Its curated Main and Diamond subsets contain 448 and 198 questions.
- Primary metric
- Accuracy is the share of four-choice questions answered correctly; higher is better only when subset and supervision setup match.
- What it measures
- graduate-level scientific question answering and the gap between domain experts, skilled outsiders, and model baselines under specified resource conditions.
- Comparison rules
- Keep the subset and access-to-search setting attached. Raw expert and non-expert figures are not interchangeable with model results on a filtered subset.
- Access and licensing
- The Compendium does not reproduce GPQA questions. Dataset access and reuse remain governed by the benchmark publisher's terms.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| Domain-expert validatorsgpqa-2023:extended:expert | 65%higher, percent | accuracyhistorical human baseline | GPQA ExtendedExperts with or pursuing relevant PhDs; raw accuracy before discounting retrospectively identified clear mistakes.The paper also reports 74% after discounting clear mistakes; that adjusted figure is a different analysis. | GPQA authorsPublished 2023-11-20, reviewed 2026-07-13human-baseline, published-third-party |
| Skilled non-expert validatorsgpqa-2023:extended:nonexpert-web | 34%higher, percent | accuracyhistorical human baseline | GPQA ExtendedExperts in other domains, unrestricted web access, averaging more than 30 minutes per question.The non-expert baseline is specific to the recruited validator population and resource allowance. | GPQA authorsPublished 2023-11-20, reviewed 2026-07-13human-baseline, published-third-party |
| GPT-4gpqa-2023:extended:few-shot-cot | 38.7%higher, percent | accuracyhistorical paper result | GPQA ExtendedFew-shot chain-of-thought prompting; exact GPT-4 provider snapshot is not stated in the result table.A model family label without a dated snapshot limits reproducibility. | GPQA authorsPublished 2023-11-20, reviewed 2026-07-13original-paper, published-third-party |
| GPT-4 with searchgpqa-2023:extended:search-backoff | 39.4%higher, percent | accuracyhistorical system result | GPQA ExtendedSearch system with backoff to the few-shot chain-of-thought answer on abstention.This is a model-plus-retrieval system, not a base-model score; the paper reports a high abstention rate on filtered subsets. | GPQA authorsPublished 2023-11-20, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2GPQA Extended contains 546 expert-written multiple-choice questions in biology, physics and chemistry. Its curated Main and Diamond subsets contain 448 and 198 questions.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.