IFEval
IFEval evaluates whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.
IFEval IFEval paper v1 (2023-11-14)
Permanent entry- Task and data
- IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.
- Primary metric
- It reports strict and loose accuracy at prompt and instruction levels. Higher is better, but each of the four metrics is a separate result lineage.
- What it measures
- whether model responses satisfy automatically checkable natural-language constraints under the benchmark's instruction inventory.
- Comparison rules
- Compare the same metric level and strictness. Model responses were collected in different months, and the paper cautions that the two models differ greatly in size.
- Access and licensing
- The Compendium does not reproduce prompts or checker code; use the official paper and linked implementation under their stated terms.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-4 (November 2023 API responses)ifeval-v1:prompt:strict | 76.89%higher, percent | prompt-level strict accuracyhistorical paper result | IFEval full evaluationAPI responses collected in November 2023; exact provider snapshot and decoding parameters are not stated in Table 3.Prompt-level strict requires every instruction in a prompt to pass. | Google ResearchPublished 2023-11-14, reviewed 2026-07-13original-paper, published-third-party |
| GPT-4 (November 2023 API responses)ifeval-v1:instruction:strict | 83.57%higher, percent | instruction-level strict accuracyhistorical paper result | IFEval full evaluationAPI responses collected in November 2023; exact provider snapshot and decoding parameters are not stated in Table 3.Instruction-level strict is not comparable to prompt-level strict despite sharing responses. | Google ResearchPublished 2023-11-14, reviewed 2026-07-13original-paper, published-third-party |
| PaLM 2 S (August 2023 API responses)ifeval-v1:prompt:strict | 43.07%higher, percent | prompt-level strict accuracyhistorical paper result | IFEval full evaluationAPI responses collected in August 2023; exact snapshot and decoding parameters are not stated in Table 3.The paper cautions that PaLM 2 S and GPT-4 are not directly comparable because parameter counts differ greatly. | Google ResearchPublished 2023-11-14, reviewed 2026-07-13original-paper, published-third-party |
| PaLM 2 S (August 2023 API responses)ifeval-v1:instruction:strict | 55.76%higher, percent | instruction-level strict accuracyhistorical paper result | IFEval full evaluationAPI responses collected in August 2023; exact snapshot and decoding parameters are not stated in Table 3.Instruction-level strict is not comparable to prompt-level strict despite sharing responses. | Google ResearchPublished 2023-11-14, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2IFEval pairs roughly 500 prompts with one or more of 25 objectively verifiable instruction types such as length, formatting, keywords, and response structure.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.