LongBench v2
LongBench v2 evaluates deep understanding and reasoning over long single documents, multiple documents, code repositories, dialogues, structured data, and in-context learning material.
LongBench v2 LongBench v2 paper v2 (2025-01-03)
Permanent entry- Task and data
- LongBench v2 contains 503 multiple-choice questions over contexts ranging from 8,000 to two million words across six long-context task families.
- Primary metric
- Overall accuracy is reported in percent, with zero-shot and zero-shot chain-of-thought columns. Higher is better within the same column and truncation policy.
- What it measures
- deep understanding and reasoning over long single documents, multiple documents, code repositories, dialogues, structured data, and in-context learning material.
- Comparison rules
- Attach zero-shot versus chain-of-thought, model snapshot, context/truncation policy, and compensation rule. LongBench v1 and v2 are different datasets.
- Access and licensing
- The Compendium cites aggregate paper results only and does not reproduce long documents, questions, structured data, or answer choices.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| o1-preview-2024-09-12longbench-v2:full:cot:uncompensated | 57.7%higher, percent | overall accuracyhistorical paper result | 503-item full setZero-shot chain-of-thought; middle truncation beyond model context; invalid choices counted incorrect in the main table.The paper's compensated table uses another scoring treatment and must not be mixed with this row. | LongBench v2 authorsPublished 2025-01-03, reviewed 2026-07-13original-paper, published-third-party |
| GPT-4o-2024-08-06longbench-v2:full:zero-shot:uncompensated | 50.1%higher, percent | overall accuracyhistorical paper result | 503-item full setZero-shot column; middle truncation beyond model context; invalid choices counted incorrect in the main table.The adjacent CoT score is 51.2%; this anchor is specifically the zero-shot column. | LongBench v2 authorsPublished 2025-01-03, reviewed 2026-07-13original-paper, published-third-party |
| Human expertslongbench-v2:full:human-15-minute | 53.7%higher, percent | overall accuracyhistorical human baseline | 503-item full setExperts had a 15-minute time limit per item and could answer 'I don't know'; this occurred for eight percent of data.Time limits and recruited expertise define this baseline; it is not an unrestricted human ceiling. | LongBench v2 authorsPublished 2025-01-03, reviewed 2026-07-13human-baseline, published-third-party |
Claims and evidence
Claim ledger, 2LongBench v2 contains 503 multiple-choice questions over contexts ranging from 8,000 to two million words across six long-context task families.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.