Humanity's Last Exam — static release history

Task and data. The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.

Scoring. The paper reports answer accuracy and RMS calibration error. The anchors below show accuracy; higher accuracy is better within the fixed static release and judge protocol.

What it measures. performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.

Comparison rules. These rows bind to the 2,500-question static paper release, not a rolling HLE dataset or later leaderboard. Text-only model rows are not comparable to full multimodal rows.

Access and licensing. The official repository uses an MIT license for repository materials and explicitly asks that benchmark data never enter training corpora. The Compendium links only and reproduces no questions or images.

Task and data. The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.

Scoring. The paper reports answer accuracy and RMS calibration error. The anchors below show accuracy; higher accuracy is better within the fixed static release and judge protocol.

What it measures. performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.

Comparison rules. These rows bind to the 2,500-question static paper release, not a rolling HLE dataset or later leaderboard. Text-only model rows are not comparable to full multimodal rows.

Access and licensing. The official repository uses an MIT license for repository materials and explicitly asks that benchmark data never enter training corpora. The Compendium links only and reproduces no questions or images.

Task and data. The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.

Scoring. The paper reports answer accuracy and RMS calibration error. The anchors below show accuracy; higher accuracy is better within the fixed static release and judge protocol.

What it measures. performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.

Comparison rules. These rows bind to the 2,500-question static paper release, not a rolling HLE dataset or later leaderboard. Text-only model rows are not comparable to full multimodal rows.

Access and licensing. The official repository uses an MIT license for repository materials and explicitly asks that benchmark data never enter training corpora. The Compendium links only and reproduces no questions or images.

Task and data. The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.

Scoring. The paper reports answer accuracy and RMS calibration error. The anchors below show accuracy; higher accuracy is better within the fixed static release and judge protocol.

What it measures. performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.

What it does not measure. open-ended research, practical work, general assistant usefulness, safety, cost, latency, or mastery of every discipline represented by one aggregate score.

Limitations. The collection filtered for questions that frontier systems missed, uses a model judge for answer equivalence, and retains a private set; near-floor differences should not be overinterpreted.

Comparison rules. These rows bind to the 2,500-question static paper release, not a rolling HLE dataset or later leaderboard. Text-only model rows are not comparable to full multimodal rows.

Access and licensing. The official repository uses an MIT license for repository materials and explicitly asks that benchmark data never enter training corpora. The Compendium links only and reproduces no questions or images.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.