LongBench v2 history

Task and data. LongBench v2 contains 503 multiple-choice questions over contexts ranging from 8,000 to two million words across six long-context task families.

Scoring. Overall accuracy is reported in percent, with zero-shot and zero-shot chain-of-thought columns. Higher is better within the same column and truncation policy.

What it measures. deep understanding and reasoning over long single documents, multiple documents, code repositories, dialogues, structured data, and in-context learning material.

Comparison rules. Attach zero-shot versus chain-of-thought, model snapshot, context/truncation policy, and compensation rule. LongBench v1 and v2 are different datasets.

Access and licensing. The Compendium cites aggregate paper results only and does not reproduce long documents, questions, structured data, or answer choices.

Task and data. LongBench v2 contains 503 multiple-choice questions over contexts ranging from 8,000 to two million words across six long-context task families.

Scoring. Overall accuracy is reported in percent, with zero-shot and zero-shot chain-of-thought columns. Higher is better within the same column and truncation policy.

What it measures. deep understanding and reasoning over long single documents, multiple documents, code repositories, dialogues, structured data, and in-context learning material.

Comparison rules. Attach zero-shot versus chain-of-thought, model snapshot, context/truncation policy, and compensation rule. LongBench v1 and v2 are different datasets.

Access and licensing. The Compendium cites aggregate paper results only and does not reproduce long documents, questions, structured data, or answer choices.

Task and data. LongBench v2 contains 503 multiple-choice questions over contexts ranging from 8,000 to two million words across six long-context task families.

Scoring. Overall accuracy is reported in percent, with zero-shot and zero-shot chain-of-thought columns. Higher is better within the same column and truncation policy.

What it measures. deep understanding and reasoning over long single documents, multiple documents, code repositories, dialogues, structured data, and in-context learning material.

What it does not measure. latency, retrieval cost, factuality beyond the answer key, production memory, open-ended synthesis, or reliable handling of every long-context distribution.

Limitations. The authors estimate about a three-percent data error rate; task mix differs by length bucket, and models use middle truncation beyond their context limits.

Comparison rules. Attach zero-shot versus chain-of-thought, model snapshot, context/truncation policy, and compensation rule. LongBench v1 and v2 are different datasets.

Access and licensing. The Compendium cites aggregate paper results only and does not reproduce long documents, questions, structured data, or answer choices.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.