GPQA history

Task and data. GPQA Extended contains 546 expert-written multiple-choice questions in biology, physics and chemistry. Its curated Main and Diamond subsets contain 448 and 198 questions.

Scoring. Accuracy is the share of four-choice questions answered correctly; higher is better only when subset and supervision setup match.

What it measures. graduate-level scientific question answering and the gap between domain experts, skilled outsiders, and model baselines under specified resource conditions.

Comparison rules. Keep the subset and access-to-search setting attached. Raw expert and non-expert figures are not interchangeable with model results on a filtered subset.

Access and licensing. The Compendium does not reproduce GPQA questions. Dataset access and reuse remain governed by the benchmark publisher's terms.

Task and data. GPQA Extended contains 546 expert-written multiple-choice questions in biology, physics and chemistry. Its curated Main and Diamond subsets contain 448 and 198 questions.

Scoring. Accuracy is the share of four-choice questions answered correctly; higher is better only when subset and supervision setup match.

What it measures. graduate-level scientific question answering and the gap between domain experts, skilled outsiders, and model baselines under specified resource conditions.

Comparison rules. Keep the subset and access-to-search setting attached. Raw expert and non-expert figures are not interchangeable with model results on a filtered subset.

Access and licensing. The Compendium does not reproduce GPQA questions. Dataset access and reuse remain governed by the benchmark publisher's terms.

Task and data. GPQA contains 448 expert-written multiple-choice questions in biology, physics, and chemistry, with Extended, Main, and Diamond subsets created through expert and non-expert validation.

Scoring. Accuracy is the share of four-choice questions answered correctly; higher is better only when subset and supervision setup match.

What it measures. graduate-level scientific question answering and the gap between domain experts, skilled outsiders, and model baselines under specified resource conditions.

What it does not measure. research creativity, laboratory ability, broad general knowledge, safety, calibration by itself, or whether an answer's reasoning is scientifically sound when the choice is correct.

Limitations. Question selection creates different difficulty and selection effects across Extended, Main, and Diamond; small differences can be noisy and public data can be contaminated.

Comparison rules. Keep the subset and access-to-search setting attached. Raw expert and non-expert figures are not interchangeable with model results on a filtered subset.

Access and licensing. The Compendium does not reproduce GPQA questions. Dataset access and reuse remain governed by the benchmark publisher's terms.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.