IFEval history

Task and data. IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.

Scoring. It reports strict and loose accuracy at prompt and instruction levels. Higher is better, but each of the four metrics is a separate result lineage.

What it measures. whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.

Comparison rules. Compare the same metric level and strictness. Model responses were collected in different months, and the paper cautions that the two models differ greatly in size.

Access and licensing. The Compendium does not reproduce prompts or checker code; use the official paper and linked implementation under their stated terms.

Task and data. IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.

Scoring. It reports strict and loose accuracy at prompt and instruction levels. Higher is better, but each of the four metrics is a separate result lineage.

What it measures. whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.

Comparison rules. Compare the same metric level and strictness. Model responses were collected in different months, and the paper cautions that the two models differ greatly in size.

Access and licensing. The Compendium does not reproduce prompts or checker code; use the official paper and linked implementation under their stated terms.

Task and data. IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.

Scoring. It reports strict and loose accuracy at prompt and instruction levels. Higher is better, but each of the four metrics is a separate result lineage.

What it measures. whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.

Comparison rules. Compare the same metric level and strictness. Model responses were collected in different months, and the paper cautions that the two models differ greatly in size.

Access and licensing. The Compendium does not reproduce prompts or checker code; use the official paper and linked implementation under their stated terms.

Task and data. IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.

Scoring. It reports strict and loose accuracy at prompt and instruction levels. Higher is better, but each of the four metrics is a separate result lineage.

What it measures. whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.

What it does not measure. helpfulness, correctness, safety, nuanced intent-following, multimodal instruction following, or constraints that cannot be verified by deterministic rules.

Limitations. Synthetic instruction combinations can be unnatural; strict checks have false negatives, while loose transformations can introduce false positives.

Comparison rules. Compare the same metric level and strictness. Model responses were collected in different months, and the paper cautions that the two models differ greatly in size.

Access and licensing. The Compendium does not reproduce prompts or checker code; use the official paper and linked implementation under their stated terms.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.