GSM8K history

Task and data. GSM8K is an 8,500-problem collection of linguistically varied grade-school arithmetic word problems whose natural-language solutions typically require several calculation steps.

Scoring. The release reports final-answer solve rate. Higher is better when the same sample, candidate count, and answer extraction rules are retained.

What it measures. multi-step grade-school mathematical word-problem solving and, in the original work, the value of ranking generated solutions with a trained verifier.

Comparison rules. Keep the 100-candidate verifier setup separate from single-generation results and modern provider-reported GSM8K variants.

Access and licensing. The Compendium links to the official release and does not reproduce word problems or solutions. Check the dataset repository before reuse.

Task and data. GSM8K is an 8,500-problem collection of linguistically varied grade-school arithmetic word problems whose natural-language solutions typically require several calculation steps.

Scoring. The release reports final-answer solve rate. Higher is better when the same sample, candidate count, and answer extraction rules are retained.

What it measures. multi-step grade-school mathematical word-problem solving and, in the original work, the value of ranking generated solutions with a trained verifier.

Comparison rules. Keep the 100-candidate verifier setup separate from single-generation results and modern provider-reported GSM8K variants.

Access and licensing. The Compendium links to the official release and does not reproduce word problems or solutions. Check the dataset repository before reuse.

Task and data. GSM8K is an 8,500-problem collection of linguistically varied grade-school arithmetic word problems whose natural-language solutions typically require several calculation steps.

Scoring. The release reports final-answer solve rate. Higher is better when the same sample, candidate count, and answer extraction rules are retained.

What it measures. multi-step grade-school mathematical word-problem solving and, in the original work, the value of ranking generated solutions with a trained verifier.

Comparison rules. Keep the 100-candidate verifier setup separate from single-generation results and modern provider-reported GSM8K variants.

Access and licensing. The Compendium links to the official release and does not reproduce word problems or solutions. Check the dataset repository before reuse.

Task and data. GSM8K is an 8,500-problem collection of linguistically varied grade-school arithmetic word problems whose natural-language solutions typically require several calculation steps.

Scoring. The release reports final-answer solve rate. Higher is better when the same sample, candidate count, and answer extraction rules are retained.

What it measures. multi-step grade-school mathematical word-problem solving and, in the original work, the value of ranking generated solutions with a trained verifier.

What it does not measure. advanced mathematics, proof validity, broad reasoning, robustness to adversarial wording, tutoring quality, or whether a correct final answer came from valid intermediate reasoning.

Limitations. A solver can reach the correct answer through flawed reasoning; the public static dataset is contamination-prone, and sampling/ranking budgets change performance.

Comparison rules. Keep the 100-candidate verifier setup separate from single-generation results and modern provider-reported GSM8K variants.

Access and licensing. The Compendium links to the official release and does not reproduce word problems or solutions. Check the dataset repository before reuse.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.