HarmBench
HarmBench evaluates comparative robustness of model defenses and effectiveness of red-team methods on the benchmark's specified harmful behavior classes.
HarmBench HarmBench paper v2 (2024-02-27)
Permanent entry- Task and data
- HarmBench standardizes harmful-behavior prompts, automated red-team attacks, target models, classifiers, and robust-refusal evaluation across text and multimodal settings.
- Primary metric
- Attack success rate is the percentage of behaviors elicited under a named attack; lower indicates stronger refusal for a target model, while higher can indicate a stronger attack.
- What it measures
- comparative robustness of model defenses and effectiveness of red-team methods on the benchmark's specified harmful behavior classes.
- Comparison rules
- The anchors below treat the subject as the target model, so lower direct-request ASR is better. Never mix Standard, Contextual, Copyright, All, validation, or test behavior splits.
- Access and licensing
- The Compendium reports aggregate safety-evaluation metadata only and reproduces no harmful prompts, target strings, jailbreaks, or attack payloads.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| Llama 2 7B Chatharmbench-v2:validation:all:direct-request | 0.8%lower, percent | direct-request attack success ratehistorical safety result | all validation behaviors: Standard, Contextual, and CopyrightDirect Request baseline with HarmBench classifier; paper Table 6 all-behavior validation aggregate.Lower is favorable only because the row's subject is the target model, not the red-team method. | HarmBench authorsPublished 2024-02-27, reviewed 2026-07-13original-paper, published-third-party |
| GPT-4 0613harmbench-v2:validation:all:direct-request | 21.0%lower, percent | direct-request attack success ratehistorical safety result | all validation behaviors: Standard, Contextual, and CopyrightDirect Request baseline with HarmBench classifier; paper Table 6 all-behavior validation aggregate.This dated API model and benchmark classifier do not represent current deployment safety. | HarmBench authorsPublished 2024-02-27, reviewed 2026-07-13original-paper, published-third-party |
| Claude 2harmbench-v2:validation:all:direct-request | 2.0%lower, percent | direct-request attack success ratehistorical safety result | all validation behaviors: Standard, Contextual, and CopyrightDirect Request baseline with HarmBench classifier; paper Table 6 all-behavior validation aggregate.The model label lacks a dated provider snapshot; no version is inferred. | HarmBench authorsPublished 2024-02-27, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2HarmBench standardizes harmful-behavior prompts, automated red-team attacks, target models, classifiers, and robust-refusal evaluation across text and multimodal settings.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.