TruthfulQA history

Task and data. TruthfulQA comprises 817 adversarially written questions in 38 categories chosen to elicit common misconceptions and imitative falsehoods from language models.

Scoring. The generation task reports truthfulness and informativeness under human or learned judgments; the multiple-choice tasks use separate accuracy metrics.

What it measures. resistance to producing selected common falsehoods when answering short questions, with a deliberate distinction between truth and informativeness.

Comparison rules. Keep generation truthfulness, true-and-informative, MC1, and MC2 separate. These anchors are original generation-task results, not the January 2025 binary variant.

Access and licensing. The Compendium publishes no benchmark questions or reference answers and links to the primary paper for methodology.

Task and data. TruthfulQA comprises 817 adversarially written questions in 38 categories chosen to elicit common misconceptions and imitative falsehoods from language models.

Scoring. The generation task reports truthfulness and informativeness under human or learned judgments; the multiple-choice tasks use separate accuracy metrics.

What it measures. resistance to producing selected common falsehoods when answering short questions, with a deliberate distinction between truth and informativeness.

Comparison rules. Keep generation truthfulness, true-and-informative, MC1, and MC2 separate. These anchors are original generation-task results, not the January 2025 binary variant.

Access and licensing. The Compendium publishes no benchmark questions or reference answers and links to the primary paper for methodology.

Task and data. TruthfulQA comprises 817 adversarially written questions in 38 categories chosen to elicit common misconceptions and imitative falsehoods from language models.

Scoring. The generation task reports truthfulness and informativeness under human or learned judgments; the multiple-choice tasks use separate accuracy metrics.

What it measures. resistance to producing selected common falsehoods when answering short questions, with a deliberate distinction between truth and informativeness.

What it does not measure. general factuality, source citation quality, real-time knowledge, calibrated uncertainty, harmlessness, or correctness across all question distributions.

Limitations. Questions were adversarially filtered against studied models, judgments can be subjective, and later binary multiple-choice revisions are not the original task.

Comparison rules. Keep generation truthfulness, true-and-informative, MC1, and MC2 separate. These anchors are original generation-task results, not the January 2025 binary variant.

Access and licensing. The Compendium publishes no benchmark questions or reference answers and links to the primary paper for methodology.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.