MMMU
MMMU evaluates college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.
MMMU MMMU paper v4 (2024-06-13)
Permanent entry- Task and data
- MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.
- Primary metric
- The paper reports multiple-choice and open-answer accuracy. The anchors below use overall validation accuracy; higher is better within that 900-item split.
- What it measures
- college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.
- Comparison rules
- Keep validation and test separate and distinguish author-provided outputs from benchmark-team runs. MMMU-Pro is a separate benchmark, not a drop-in update.
- Access and licensing
- The official repository says annotators were instructed to respect source rights and invites removal reports. The Compendium reproduces no questions, textbook figures, music, charts, or tables.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| Random choicemmmu-v4:validation-900:overall | 22.1%higher, percent | accuracyhistorical baseline | 900-item validation setAnswer-frequency baseline reported in paper Table 2.The chance level is not exactly 25% because answer-option counts vary. | MMMU authorsPublished 2024-06-13, reviewed 2026-07-13original-paper, published-third-party |
| Best expert groupmmmu-v4:validation-900:human-expert | 88.6%higher, percent | accuracyhistorical human baseline | 900-item validation setBest of three college-senior expert levels; 90 students represented 30 subjects and could consult textbooks.This recruited expert comparison is not a universal human ceiling. | MMMU authorsPublished 2024-06-13, reviewed 2026-07-13human-baseline, published-third-party |
| GPT-4V(ision) (Playground)mmmu-v4:validation-900:text-image | 56.8%higher, percent | accuracyhistorical paper result | 900-item validation setText-and-image input under the benchmark paper's Playground evaluation; exact dated model snapshot is not stated.Missing exact provider snapshot limits reproduction. | MMMU authorsPublished 2024-06-13, reviewed 2026-07-13original-paper, published-third-party |
| GPT-4ommmu-v4:validation-900:author-provided | 69.1%higher, percent | accuracyhistorical author-provided result | 900-item validation setAuthor-provided result; the paper does not report one unified prompt/setup for this row.The source marks this as author-provided; absent setup details are not filled in by the Compendium. | MMMU authorsPublished 2024-06-13, reviewed 2026-07-13model-provider, published-third-party |
Claims and evidence
Claim ledger, 2MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.