MMLU history

Task and data. MMLU aggregates four-choice questions across 57 academic and professional subjects, with development, validation, and held-out test examples.

Scoring. The paper reports classification accuracy averaged across examples and tasks; higher is better within the same release and prompting protocol.

What it measures. broad text-only knowledge recall and problem solving across the sampled school, university, and professional domains.

Comparison rules. Compare only results using the same 57-task release and matching shot/scoring protocol. Later MMLU variants and provider-reported aggregates may use different prompts.

Access and licensing. The Compendium links to the paper and does not reproduce questions. Underlying question sources can retain separate rights and terms.

Task and data. MMLU aggregates four-choice questions across 57 academic and professional subjects, with development, validation, and held-out test examples.

Scoring. The paper reports classification accuracy averaged across examples and tasks; higher is better within the same release and prompting protocol.

What it measures. broad text-only knowledge recall and problem solving across the sampled school, university, and professional domains.

Comparison rules. Compare only results using the same 57-task release and matching shot/scoring protocol. Later MMLU variants and provider-reported aggregates may use different prompts.

Access and licensing. The Compendium links to the paper and does not reproduce questions. Underlying question sources can retain separate rights and terms.

Task and data. MMLU aggregates four-choice questions across 57 academic and professional subjects, with development, validation, and held-out test examples.

Scoring. The paper reports classification accuracy averaged across examples and tasks; higher is better within the same release and prompting protocol.

What it measures. broad text-only knowledge recall and problem solving across the sampled school, university, and professional domains.

What it does not measure. open-ended generation quality, multimodal perception, tool use, safety, latency, cost, or performance on a user's own domain.

Limitations. Static public questions can enter training data, subject difficulty varies, and one aggregate can conceal large domain-level weaknesses.

Comparison rules. Compare only results using the same 57-task release and matching shot/scoring protocol. Later MMLU variants and provider-reported aggregates may use different prompts.

Access and licensing. The Compendium links to the paper and does not reproduce questions. Underlying question sources can retain separate rights and terms.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.