MMMU history

Task and data. MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.

Scoring. The paper reports multiple-choice and open-answer accuracy. The anchors below use overall validation accuracy; higher is better within that 900-item split.

What it measures. college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.

Comparison rules. Keep validation and test separate and distinguish author-provided outputs from benchmark-team runs. MMMU-Pro is a separate benchmark, not a drop-in update.

Access and licensing. The official repository says annotators were instructed to respect source rights and invites removal reports. The Compendium reproduces no questions, textbook figures, music, charts, or tables.

Task and data. MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.

Scoring. The paper reports multiple-choice and open-answer accuracy. The anchors below use overall validation accuracy; higher is better within that 900-item split.

What it measures. college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.

Comparison rules. Keep validation and test separate and distinguish author-provided outputs from benchmark-team runs. MMMU-Pro is a separate benchmark, not a drop-in update.

Access and licensing. The official repository says annotators were instructed to respect source rights and invites removal reports. The Compendium reproduces no questions, textbook figures, music, charts, or tables.

Task and data. MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.

Scoring. The paper reports multiple-choice and open-answer accuracy. The anchors below use overall validation accuracy; higher is better within that 900-item split.

What it measures. college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.

Comparison rules. Keep validation and test separate and distinguish author-provided outputs from benchmark-team runs. MMMU-Pro is a separate benchmark, not a drop-in update.

Access and licensing. The official repository says annotators were instructed to respect source rights and invites removal reports. The Compendium reproduces no questions, textbook figures, music, charts, or tables.

Task and data. MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.

Scoring. The paper reports multiple-choice and open-answer accuracy. The anchors below use overall validation accuracy; higher is better within that 900-item split.

What it measures. college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.

What it does not measure. general visual intelligence, accessibility, real-world tool use, safety, cost, latency, or performance outside the benchmark's selected academic sources.

Limitations. Source heterogeneity, OCR sensitivity, answer format, prompt setup, and potential exposure to textbook or exam material affect results.

Comparison rules. Keep validation and test separate and distinguish author-provided outputs from benchmark-team runs. MMMU-Pro is a separate benchmark, not a drop-in update.

Access and licensing. The official repository says annotators were instructed to respect source rights and invites removal reports. The Compendium reproduces no questions, textbook figures, music, charts, or tables.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.