MMLU
MMLU evaluates broad text-only knowledge recall and problem solving across the sampled school, university, and professional domains.
MMLU ICLR 2021 paper release
Permanent entry- Task and data
- MMLU aggregates four-choice questions across 57 academic and professional subjects, with development, validation, and held-out test examples.
- Primary metric
- The paper reports classification accuracy averaged across examples and tasks; higher is better within the same release and prompting protocol.
- What it measures
- broad text-only knowledge recall and problem solving across the sampled school, university, and professional domains.
- Comparison rules
- Compare only results using the same 57-task release and matching shot/scoring protocol. Later MMLU variants and provider-reported aggregates may use different prompts.
- Access and licensing
- The Compendium links to the paper and does not reproduce questions. Underlying question sources can retain separate rights and terms.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-3 175B (paper label: X-Large)mmlu-2021:test:57-task:few-shot | 43.9%higher, percent | average classification accuracyhistorical paper baseline | 57-task test aggregateFew-shot prompting; the paper evaluates the 175B GPT-3 variant across all tasks.This is the original paper baseline, not a current GPT-family result. | MMLU authorsPublished 2021-01-12, reviewed 2026-07-13original-paper, published-third-party |
| Unspecialized Amazon Mechanical Turk participantsmmlu-2021:human:unspecialized | 34.5%higher, percent | average classification accuracyhistorical human baseline | human comparison over the benchmarkHuman baseline described by the paper; participant sampling detail is limited in the result summary.A crowd-worker baseline is not an estimate of expert performance. | MMLU authorsPublished 2021-01-12, reviewed 2026-07-13human-baseline, published-third-party |
| Estimated expert-level human performancemmlu-2021:human:expert-estimate | 89.8%higher, percent | estimated average accuracyhistorical estimate | 57-subject aggregate estimatePaper authors combine available 95th-percentile exam performance with estimates where equivalent data are unavailable.This is an author estimate, not one controlled participant run across every MMLU subject. | MMLU authorsPublished 2021-01-12, reviewed 2026-07-13human-baseline, published-third-party |
Claims and evidence
Claim ledger, 2MMLU aggregates four-choice questions across 57 academic and professional subjects, with development, validation, and held-out test examples.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.