LiveCodeBench
LiveCodeBench evaluates coding performance on recently released contest problems and several adjacent code-reasoning scenarios with time-window analysis for possible contamination.
LiveCodeBench LiveCodeBench paper v2; post-September-2023 slice
Permanent entry- Task and data
- LiveCodeBench continuously collects coding-contest problems after model release dates and evaluates code generation, self-repair, code execution, and output prediction.
- Primary metric
- The paper uses scenario-specific pass@1. The anchors below are total code-generation pass@1 on the paper's post-September-2023 window; higher is better.
- What it measures
- coding performance on recently released contest problems and several adjacent code-reasoning scenarios with time-window analysis for possible contamination.
- Comparison rules
- Pin benchmark release, scenario, date window, difficulty aggregation, prompt, and model snapshot. The paper-v2 anchors are not current leaderboard standings.
- Access and licensing
- The official code is MIT-licensed, but contest problem statements originate with LeetCode, AtCoder, and Codeforces and retain platform-specific terms. No problems are reproduced here.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| GPT-4O-2024-05-13livecodebench-v2:post-2023-09:codegen:total | 41.9%higher, percent | code-generation pass@1fixed paper snapshot of rolling benchmark | post-September-2023 problems; all difficulty levelsZero-shot code-generation prompt; total combines Easy, Medium, and Hard in paper Table 3.This dated paper slice is not a current rolling-leaderboard score. | LiveCodeBench authorsPublished 2024-06-06, reviewed 2026-07-13rolling-leaderboard, published-third-party |
| GPT-4-Turbo-2024-04-09livecodebench-v2:post-2023-09:codegen:total | 41.1%higher, percent | code-generation pass@1fixed paper snapshot of rolling benchmark | post-September-2023 problems; all difficulty levelsZero-shot code-generation prompt; total combines Easy, Medium, and Hard in paper Table 3.Prompt and time-window changes move this result into a different comparison class. | LiveCodeBench authorsPublished 2024-06-06, reviewed 2026-07-13rolling-leaderboard, published-third-party |
| Claude-3-Opuslivecodebench-v2:post-2023-09:codegen:total | 32.8%higher, percent | code-generation pass@1fixed paper snapshot of rolling benchmark | post-September-2023 problems; all difficulty levelsZero-shot code-generation prompt; exact provider snapshot is not stated beyond the paper label.The missing exact model snapshot limits reproducibility and is not inferred. | LiveCodeBench authorsPublished 2024-06-06, reviewed 2026-07-13rolling-leaderboard, published-third-party |
Claims and evidence
Claim ledger, 2LiveCodeBench continuously collects coding-contest problems after model release dates and evaluates code generation, self-repair, code execution, and output prediction.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.