AI benchmark index

Task and data

ARC-AGI-2 evaluates programmatic solutions on novel abstract grid-transformation tasks; the 2025 competition used a hidden private evaluation under efficiency and cost constraints.

Primary metric

A task scores as solved when one of two submitted outputs exactly matches the hidden output; the final accuracy is averaged over private tasks and higher is better.

What it measures

few-shot abstraction, rule induction, and exact generalization on unfamiliar visual grid transformations under the competition's resource constraints.

Comparison rules

Compare only the same ARC-AGI-2 year, split, submission allowance, and budget. These are 2025 competition system results, not model-only scores.

Access and licensing

Use ARC data and competition assets only under their official terms. The Compendium reproduces no grids, tasks, or solution artifacts.

Swipe or scroll sideways for every evidence field

Task and data

GAIA contains 466 real-world assistant questions divided into three difficulty levels and requiring combinations of reasoning, web browsing, files, multimodal inputs, and tools.

Primary metric

Accuracy is the share of questions receiving the unambiguous final answer; higher is better for a fixed split and tool-access policy.

What it measures

general-assistant task completion when answers require coordinated reasoning, retrieval, multimodal handling, and external tool use.

Comparison rules

A GAIA score is a system score. Match the 2023 release, split, tool set, browser state, and answer normalization before comparing results.

Access and licensing

The GAIA dataset card forbids resharing validation or test data in a crawlable form and requires gated access. The Compendium reproduces no questions, files, or answers.

Swipe or scroll sideways for every evidence field

Task and data

GPQA Extended contains 546 expert-written multiple-choice questions in biology, physics and chemistry. Its curated Main and Diamond subsets contain 448 and 198 questions.

Primary metric

Accuracy is the share of four-choice questions answered correctly; higher is better only when subset and supervision setup match.

What it measures

graduate-level scientific question answering and the gap between domain experts, skilled outsiders, and model baselines under specified resource conditions.

Comparison rules

Keep the subset and access-to-search setting attached. Raw expert and non-expert figures are not interchangeable with model results on a filtered subset.

Access and licensing

The Compendium does not reproduce GPQA questions. Dataset access and reuse remain governed by the benchmark publisher's terms.

Swipe or scroll sideways for every evidence field

Task and data

GSM8K is an 8,500-problem collection of linguistically varied grade-school arithmetic word problems whose natural-language solutions typically require several calculation steps.

Primary metric

The release reports final-answer solve rate. Higher is better when the same sample, candidate count, and answer extraction rules are retained.

What it measures

multi-step grade-school mathematical word-problem solving and, in the original work, the value of ranking generated solutions with a trained verifier.

Comparison rules

Keep the 100-candidate verifier setup separate from single-generation results and modern provider-reported GSM8K variants.

Access and licensing

The Compendium links to the official release and does not reproduce word problems or solutions. Check the dataset repository before reuse.

Swipe or scroll sideways for every evidence field

Task and data

HarmBench standardizes harmful-behavior prompts, automated red-team attacks, target models, classifiers, and robust-refusal evaluation across text and multimodal settings.

Primary metric

Attack success rate is the percentage of behaviors elicited under a named attack; lower indicates stronger refusal for a target model, while higher can indicate a stronger attack.

What it measures

comparative robustness of model defenses and effectiveness of red-team methods on the benchmark's specified harmful behavior classes.

Comparison rules

The anchors below treat the subject as the target model, so lower direct-request ASR is better. Never mix Standard, Contextual, Copyright, All, validation, or test behavior splits.

Access and licensing

The Compendium reports aggregate safety-evaluation metadata only and reproduces no harmful prompts, target strings, jailbreaks, or attack payloads.

Swipe or scroll sideways for every evidence field

Task and data

HumanEval is a set of 164 hand-written Python function-synthesis tasks defined by a signature, docstring, body placeholder, and hidden unit tests.

Primary metric

The paper reports pass@k, an estimate of the probability that at least one of k generated samples passes all tests; higher is better for a fixed k and sampling setup.

What it measures

small, self-contained Python code generation from natural-language function specifications under functional tests.

Comparison rules

Never compare pass@1 to pass@10 or pass@100 as one metric. HumanEval+, language ports, and modified test suites are separate lineages.

Access and licensing

The Compendium cites aggregate results only and does not reproduce prompts, reference code, or tests.

Swipe or scroll sideways for every evidence field

Task and data

The static HLE release contains 2,500 closed-ended, expert-reviewed text and image questions spanning more than one hundred subjects at the frontier of academic knowledge.

Primary metric

The paper reports answer accuracy and RMS calibration error. The anchors below show accuracy; higher accuracy is better within the fixed static release and judge protocol.

What it measures

performance on a deliberately difficult, broad set of expert-level closed-ended academic questions, including multimodal items.

Comparison rules

These rows bind to the 2,500-question static paper release, not a rolling HLE dataset or later leaderboard. Text-only model rows are not comparable to full multimodal rows.

Access and licensing

The official repository uses an MIT license for repository materials and explicitly asks that benchmark data never enter training corpora. The Compendium links only and reproduces no questions or images.

Swipe or scroll sideways for every evidence field

Task and data

IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.

Primary metric

It reports strict and loose accuracy at prompt and instruction levels. Higher is better, but each of the four metrics is a separate result lineage.

What it measures

whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.

Comparison rules

Compare the same metric level and strictness. Model responses were collected in different months, and the paper cautions that the two models differ greatly in size.

Access and licensing

The Compendium does not reproduce prompts or checker code; use the official paper and linked implementation under their stated terms.

Swipe or scroll sideways for every evidence field

Task and data

LiveCodeBench continuously collects coding-contest problems after model release dates and evaluates code generation, self-repair, code execution, and output prediction.

Primary metric

The paper uses scenario-specific pass@1. The anchors below are total code-generation pass@1 on the paper's post-September-2023 window; higher is better.

What it measures

coding performance on recently released contest problems and several adjacent code-reasoning scenarios with time-window analysis for possible contamination.

Comparison rules

Pin benchmark release, scenario, date window, difficulty aggregation, prompt, and model snapshot. The paper-v2 anchors are not current leaderboard standings.

Access and licensing

The official code is MIT-licensed, but contest problem statements originate with LeetCode, AtCoder, and Codeforces and retain platform-specific terms. No problems are reproduced here.

Swipe or scroll sideways for every evidence field

Task and data

LongBench v2 contains 503 multiple-choice questions over contexts ranging from 8,000 to two million words across six long-context task families.

Primary metric

Overall accuracy is reported in percent, with zero-shot and zero-shot chain-of-thought columns. Higher is better within the same column and truncation policy.

What it measures

deep understanding and reasoning over long single documents, multiple documents, code repositories, dialogues, structured data, and in-context learning material.

Comparison rules

Attach zero-shot versus chain-of-thought, model snapshot, context/truncation policy, and compensation rule. LongBench v1 and v2 are different datasets.

Access and licensing

The Compendium cites aggregate paper results only and does not reproduce long documents, questions, structured data, or answer choices.

Swipe or scroll sideways for every evidence field

Task and data

MATH contains 12,500 competition-mathematics problems across seven subjects and five difficulty levels, each paired with a worked solution in the released dataset.

Primary metric

The paper primarily reports exact-answer accuracy after normalization of equivalent final-answer formats; higher is better within the same split and generation rules.

What it measures

competition-style mathematical problem solving across algebra, geometry, number theory, counting and probability, and related school-level areas.

Comparison rules

Separate few-shot GPT-3 from AMPS-pretrained and fine-tuned GPT-2 results. Modern MATH variants and scoring pipelines are different lineages.

Access and licensing

The Compendium does not reproduce problems, figures, or solutions. Code licensing does not automatically grant rights to every source problem.

Swipe or scroll sideways for every evidence field

Task and data

MMLU aggregates four-choice questions across 57 academic and professional subjects, with development, validation, and held-out test examples.

Primary metric

The paper reports classification accuracy averaged across examples and tasks; higher is better within the same release and prompting protocol.

What it measures

broad text-only knowledge recall and problem solving across the sampled school, university, and professional domains.

Comparison rules

Compare only results using the same 57-task release and matching shot/scoring protocol. Later MMLU variants and provider-reported aggregates may use different prompts.

Access and licensing

The Compendium links to the paper and does not reproduce questions. Underlying question sources can retain separate rights and terms.

Swipe or scroll sideways for every evidence field

Task and data

MMLU-Pro revises MMLU into 14 reasoning-heavy categories, removes noisy or trivial items, and expands multiple-choice questions from four to ten options.

Primary metric

The paper reports overall accuracy in percent. Its main table uses chain-of-thought and five shots except for the two Gemini 1.5 rows, which use zero shots.

What it measures

more discriminative knowledge-and-reasoning performance under the paper's ten-option, curated multiple-choice construction.

Comparison rules

Do not compare the rows below to ordinary MMLU scores. Within MMLU-Pro, preserve the paper version and distinguish five-shot from zero-shot settings.

Access and licensing

The Compendium paraphrases paper metadata and does not reproduce benchmark questions or answer choices.

Swipe or scroll sideways for every evidence field

Task and data

MMMU contains about 11,500 multimodal questions from college exams, quizzes, and textbooks across 30 subjects, 183 subfields, and many image types.

Primary metric

The paper reports multiple-choice and open-answer accuracy. The anchors below use overall validation accuracy; higher is better within that 900-item split.

What it measures

college-level multimodal perception plus domain knowledge and reasoning across art, business, science, medicine, humanities, and engineering.

Comparison rules

Keep validation and test separate and distinguish author-provided outputs from benchmark-team runs. MMMU-Pro is a separate benchmark, not a drop-in update.

Access and licensing

The official repository says annotators were instructed to respect source rights and invites removal reports. The Compendium reproduces no questions, textbook figures, music, charts, or tables.

Swipe or scroll sideways for every evidence field

Task and data

MTEB v1 spans eight embedding task types; the paper's English aggregate covers 56 datasets across classification, clustering, pair classification, reranking, retrieval, semantic similarity, and summarization.

Primary metric

Each task type has its own principal metric; the paper's overall score averages task-level main metrics on a 0–100 scale. Higher is better within the fixed dataset suite.

What it measures

how one text-embedding model transfers across several retrieval, similarity, classification, clustering, reranking, and related English tasks.

Comparison rules

These are MTEB v1 paper aggregates over 56 English datasets. Do not merge them with current MTEB leaderboard variants, multilingual suites, or different task weighting.

Access and licensing

The benchmark framework and each underlying dataset can have different licenses. The Compendium reproduces no examples or dataset content.

Swipe or scroll sideways for every evidence field

Task and data

OSWorld provides 369 open-ended computer tasks in real Ubuntu, Windows, and macOS environments across web, office, media, coding, and cross-application workflows.

Primary metric

Execution-based success rate is the share of tasks whose environment end state satisfies the task-specific evaluator; higher is better under the same OS and observation/action setup.

What it measures

multimodal computer-agent ability to perceive interfaces and execute mouse, keyboard, and code actions across real desktop applications.

Comparison rules

Preserve input modality, action space, operating-system image, model snapshot, and task release. Human, accessibility-tree, screenshot, and Set-of-Mark rows are distinct.

Access and licensing

The Compendium does not reproduce tasks, screenshots, application assets, or environment images; use official access paths and third-party software licenses.

Swipe or scroll sideways for every evidence field

Task and data

SWE-Bench Pro contains 1,865 longer-horizon software-engineering tasks across 41 maintained public, held-out, and commercial repositories, with human-augmented requirements and reproducible tests.

Primary metric

Percent resolved is the fraction of tasks whose patches satisfy the benchmark tests. Higher is better only within the same public, held-out, or commercial split and scaffold.

What it measures

agentic repository work on larger, more diverse, and less easily memorized software tasks than the original SWE-bench family.

Comparison rules

Never pool public and commercial results. These release-page anchors use the SWE-Agent scaffold and model labels exactly as published; exact provider snapshots are missing.

Access and licensing

Public, held-out, and commercial materials have different access and licensing conditions. The Compendium publishes metadata and results only, not tasks or code.

Swipe or scroll sideways for every evidence field

Task and data

SWE-bench Verified is a human-filtered set of 500 real GitHub issue-resolution tasks whose problem statements, tests, and environments were reviewed for feasibility.

Primary metric

Percent resolved is the share of instances whose generated patch applies and passes both FAIL_TO_PASS and PASS_TO_PASS tests; higher is better for a fixed scaffold and harness.

What it measures

agent-plus-model ability to navigate Python repositories and produce test-passing patches for selected real-world issues.

Comparison rules

Treat the 2024 OpenAI figure as historical. Current mini-SWE-agent releases 1.x and 2.x are explicitly not necessarily comparable, and arbitrary-scaffold results form other lineages.

Access and licensing

Repository issues, code, and tests retain their original project licenses. The Compendium does not reproduce issue text, patches, tests, or repository assets.

Swipe or scroll sideways for every evidence field

Task and data

TruthfulQA comprises 817 adversarially written questions in 38 categories chosen to elicit common misconceptions and imitative falsehoods from language models.

Primary metric

The generation task reports truthfulness and informativeness under human or learned judgments; the multiple-choice tasks use separate accuracy metrics.

What it measures

resistance to producing selected common falsehoods when answering short questions, with a deliberate distinction between truth and informativeness.

Comparison rules

Keep generation truthfulness, true-and-informative, MC1, and MC2 separate. These anchors are original generation-task results, not the January 2025 binary variant.

Access and licensing

The Compendium publishes no benchmark questions or reference answers and links to the primary paper for methodology.

Swipe or scroll sideways for every evidence field