Humanity's Last Exam — static release
Humanity's Last Exam — static release evaluates performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.
Humanity's Last Exam — static release 2,500-question static release; paper v10
Permanent entry- Task and data
- The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.
- Primary metric
- The paper reports answer accuracy and RMS calibration error. The anchors below show accuracy; higher accuracy is better within the fixed static release and judge protocol.
- What it measures
- performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.
- Comparison rules
- These rows bind to the 2,500-question static paper release, not a rolling HLE dataset or later leaderboard. Text-only model rows are not comparable to full multimodal rows.
- Access and licensing
- The official repository uses an MIT license for repository materials and explicitly asks that benchmark data never enter training corpora. The Compendium links only and reproduces no questions or images.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| gpt-4o-2024-11-20hle-static-2500:full:paper-v10 | 2.7%higher, percent | accuracystatic historical anchor | 2,500-question static releaseStandardized zero-shot reasoning/final-answer prompt; temperature 0; o3-mini answer-equivalence judge.The static-release score should not be relabeled as a current HLE leaderboard result. | HLE organizing teamPublished 2026-02-20, reviewed 2026-07-13original-paper, published-third-party |
| claude-3-5-sonnet-20241022hle-static-2500:full:paper-v10 | 4.1%higher, percent | accuracystatic historical anchor | 2,500-question static releaseStandardized zero-shot reasoning/final-answer prompt; temperature 0; o3-mini answer-equivalence judge.Small near-floor differences are not strong evidence of broad capability gains, as the paper cautions. | HLE organizing teamPublished 2026-02-20, reviewed 2026-07-13original-paper, published-third-party |
| o1-2024-12-17hle-static-2500:full:paper-v10 | 8.0%higher, percent | accuracystatic historical anchor | 2,500-question static releaseStandardized zero-shot reasoning/final-answer prompt; model-supported temperature 1.0; o3-mini answer-equivalence judge.Different reasoning budgets and model-specific temperature constraints matter to comparison. | HLE organizing teamPublished 2026-02-20, reviewed 2026-07-13original-paper, published-third-party |
| o3-mini-2025-01-31 (high)hle-static-2500:text-only:paper-v10 | 13.4%higher, percent | accuracystatic subset anchor | text-only subset of the static releaseStandardized zero-shot reasoning/final-answer prompt; model-supported temperature 1.0; o3-mini answer-equivalence judge.The paper marks this model as text-only, so its subset result is not directly comparable to full multimodal rows. | HLE organizing teamPublished 2026-02-20, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.