TruthfulQA
TruthfulQA evaluates resistance to producing selected common falsehoods when answering short questions, with a deliberate distinction between truth and informativeness.
TruthfulQA TruthfulQA paper v2 (2022-05-08)
Permanent entry- Task and data
- TruthfulQA comprises 817 adversarially written questions in 38 categories chosen to elicit common misconceptions and imitative falsehoods from language models.
- Primary metric
- The generation task reports truthfulness and informativeness under human or learned judgments; the multiple-choice tasks use separate accuracy metrics.
- What it measures
- resistance to producing selected common falsehoods when answering short questions, with a deliberate distinction between truth and informativeness.
- Comparison rules
- Keep generation truthfulness, true-and-informative, MC1, and MC2 separate. These anchors are original generation-task results, not the January 2025 binary variant.
- Access and licensing
- The Compendium publishes no benchmark questions or reference answers and links to the primary paper for methodology.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-3 175B with helpful prompttruthfulqa-v2:generation:truthfulness | 58%higher, percent | truthful answershistorical paper result | 817-question generation taskHelpful prompt; generated full-sentence answers judged for truthfulness under the paper's protocol.Only 21% of answers were both true and informative; truthfulness alone can reward uninformative responses. | TruthfulQA authorsPublished 2022-05-08, reviewed 2026-07-13original-paper, published-third-party |
| Human participanttruthfulqa-v2:generation:truthfulness:human | 94%higher, percent | truthful answershistorical human baseline | 817-question generation taskSingle human participant answered the generation questions under the paper's comparison protocol.One participant is an illustrative anchor, not a population estimate; 87% of answers were both true and informative. | TruthfulQA authorsPublished 2022-05-08, reviewed 2026-07-13human-baseline, published-third-party |
Claims and evidence
Claim ledger, 2TruthfulQA comprises 817 adversarially written questions in 38 categories chosen to elicit common misconceptions and imitative falsehoods from language models.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.