HarmBench history

Task and data. HarmBench standardizes harmful-behavior prompts, automated red-team attacks, target models, classifiers, and robust-refusal evaluation across text and multimodal settings.

Scoring. Attack success rate is the percentage of behaviors elicited under a named attack; lower indicates stronger refusal for a target model, while higher can indicate a stronger attack.

What it measures. comparative robustness of model defenses and effectiveness of red-team methods on the benchmark's specified harmful behavior classes.

Comparison rules. The anchors below treat the subject as the target model, so lower direct-request ASR is better. Never mix Standard, Contextual, Copyright, All, validation, or test behavior splits.

Access and licensing. The Compendium reports aggregate safety-evaluation metadata only and reproduces no harmful prompts, target strings, jailbreaks, or attack payloads.

Task and data. HarmBench standardizes harmful-behavior prompts, automated red-team attacks, target models, classifiers, and robust-refusal evaluation across text and multimodal settings.

Scoring. Attack success rate is the percentage of behaviors elicited under a named attack; lower indicates stronger refusal for a target model, while higher can indicate a stronger attack.

What it measures. comparative robustness of model defenses and effectiveness of red-team methods on the benchmark's specified harmful behavior classes.

Comparison rules. The anchors below treat the subject as the target model, so lower direct-request ASR is better. Never mix Standard, Contextual, Copyright, All, validation, or test behavior splits.

Access and licensing. The Compendium reports aggregate safety-evaluation metadata only and reproduces no harmful prompts, target strings, jailbreaks, or attack payloads.

Task and data. HarmBench standardizes harmful-behavior prompts, automated red-team attacks, target models, classifiers, and robust-refusal evaluation across text and multimodal settings.

Scoring. Attack success rate is the percentage of behaviors elicited under a named attack; lower indicates stronger refusal for a target model, while higher can indicate a stronger attack.

What it measures. comparative robustness of model defenses and effectiveness of red-team methods on the benchmark's specified harmful behavior classes.

What it does not measure. complete real-world safety, benign refusal quality, every abuse domain, production monitoring, jailbreak cost, or safety under untested system prompts and tools.

Limitations. Classifier judgments can err, harmful behavior taxonomies are incomplete, attack budgets differ, and the meaning of direction depends on whether the subject is a model or attack.

Comparison rules. The anchors below treat the subject as the target model, so lower direct-request ASR is better. Never mix Standard, Contextual, Copyright, All, validation, or test behavior splits.

Access and licensing. The Compendium reports aggregate safety-evaluation metadata only and reproduces no harmful prompts, target strings, jailbreaks, or attack payloads.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.