MMLU-Pro history

Task and data. MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.

Scoring. The paper reports overall accuracy in percent. Its main table uses chain-of-thought and five shots except for the two Gemini 1.5 rows, which use zero shots.

What it measures. more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.

Comparison rules. Do not compare the rows below to ordinary MMLU scores. Within MMLU-Pro, preserve the paper version and distinguish five-shot from zero-shot settings.

Access and licensing. The Compendium paraphrases paper metadata and does not reproduce benchmark questions or answer choices.

Task and data. MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.

Scoring. The paper reports overall accuracy in percent. Its main table uses chain-of-thought and five shots except for the two Gemini 1.5 rows, which use zero shots.

What it measures. more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.

Comparison rules. Do not compare the rows below to ordinary MMLU scores. Within MMLU-Pro, preserve the paper version and distinguish five-shot from zero-shot settings.

Access and licensing. The Compendium paraphrases paper metadata and does not reproduce benchmark questions or answer choices.

Task and data. MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.

Scoring. The paper reports overall accuracy in percent. Its main table uses chain-of-thought and five shots except for the two Gemini 1.5 rows, which use zero shots.

What it measures. more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.

Comparison rules. Do not compare the rows below to ordinary MMLU scores. Within MMLU-Pro, preserve the paper version and distinguish five-shot from zero-shot settings.

Access and licensing. The Compendium paraphrases paper metadata and does not reproduce benchmark questions or answer choices.

Task and data. MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.

Scoring. The paper reports overall accuracy in percent. Its main table uses chain-of-thought and five shots except for the two Gemini 1.5 rows, which use zero shots.

What it measures. more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.

What it does not measure. agent reliability, factuality outside selected domains, multimodal understanding, safety behavior, inference efficiency, or real-workflow outcomes.

Limitations. The benchmark remains a static academic test; chain-of-thought availability, prompt wording, contamination, and category mix affect the aggregate.

Comparison rules. Do not compare the rows below to ordinary MMLU scores. Within MMLU-Pro, preserve the paper version and distinguish five-shot from zero-shot settings.

Access and licensing. The Compendium paraphrases paper metadata and does not reproduce benchmark questions or answer choices.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.