MMLU-Pro
MMLU-Pro evaluates more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.
MMLU-Pro paper v6 (2024-11-06)
Permanent entry- Task and data
- MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.
- Primary metric
- The paper reports overall accuracy in percent. Its main table uses chain-of-thought and five shots except for the two Gemini 1.5 rows, which use zero shots.
- What it measures
- more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.
- Comparison rules
- Do not compare the rows below to ordinary MMLU scores. Within MMLU-Pro, preserve the paper version and distinguish five-shot from zero-shot settings.
- Access and licensing
- The Compendium paraphrases paper metadata and does not reproduce benchmark questions or answer choices.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-4o (paper's June 2024 evaluation label)mmlu-pro-v6:test:cot:5-shot | 72.6%higher, percent | overall accuracyhistorical paper result | overall test aggregateChain-of-thought, five-shot setting from Table 2; exact provider snapshot is not stated in the table.The exact dated GPT-4o API model is not reported, so this row must not be merged with provider-version results. | MMLU-Pro authorsPublished 2024-11-06, reviewed 2026-07-13original-paper, published-third-party |
| Gemini-1.5-Prommlu-pro-v6:test:cot:0-shot | 69.0%higher, percent | overall accuracyhistorical paper result | overall test aggregateChain-of-thought, zero-shot setting from Table 2; exact provider snapshot is not stated in the table.This row uses zero shots while most Table 2 rows use five, so direct model comparison needs that qualification. | MMLU-Pro authorsPublished 2024-11-06, reviewed 2026-07-13original-paper, published-third-party |
| Claude-3-Opusmmlu-pro-v6:test:cot:5-shot | 68.5%higher, percent | overall accuracyhistorical paper result | overall test aggregateChain-of-thought, five-shot setting from Table 2; exact provider snapshot is not stated in the table.Provider-side model updates and prompt differences can invalidate comparisons to similarly named later snapshots. | MMLU-Pro authorsPublished 2024-11-06, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.