OSWorld
OSWorld evaluates multimodal computer-agent ability to perceive interfaces and execute mouse, keyboard, and code actions across real desktop applications.
OSWorld OSWorld paper v2 (2024-05-30)
Permanent entry- Task and data
- OSWorld provides 369 open-ended computer tasks in real Ubuntu, Windows, and macOS environments across web, office, media, coding, and cross-application workflows.
- Primary metric
- Execution-based success rate is the share of tasks whose environment end state satisfies the task-specific evaluator; higher is better under the same OS and observation/action setup.
- What it measures
- multimodal computer-agent ability to perceive interfaces and execute mouse, keyboard, and code actions across real desktop applications.
- Comparison rules
- Preserve input modality, action space, operating-system image, model snapshot, and task release. Human, accessibility-tree, screenshot, and Set-of-Mark rows are distinct.
- Access and licensing
- The Compendium does not reproduce tasks, screenshots, application assets, or environment images; use official access paths and third-party software licenses.
| Published subject | Score | Metric | Split and evaluation setting | Source record |
|---|---|---|---|---|
| Human participantsosworld-v2:full:human | 72.36%higher, percent | execution success ratehistorical human baseline | OSWorld full task setPaper human-performance study using real environments and task-specific execution checks.Human performance reflects the study's participants, environment, and time conditions. | OSWorld authorsPublished 2024-05-30, reviewed 2026-07-13human-baseline, published-third-party |
| GPT-4osworld-v2:full:a11y-tree | 12.24%higher, percent | execution success ratehistorical agent result | OSWorld full task setAccessibility-tree input and pyautogui-style computer actions; three recent observations/actions in chat history.The paper label does not supply an exact dated GPT-4 snapshot; no version is inferred. | OSWorld authorsPublished 2024-05-30, reviewed 2026-07-13original-paper, published-third-party |
| GPT-4oosworld-v2:full:a11y-tree | 11.36%higher, percent | execution success ratehistorical agent result | OSWorld full task setAccessibility-tree input and pyautogui-style computer actions; three recent observations/actions in chat history.This is one observation/action configuration, not a general GPT-4o computer-use score. | OSWorld authorsPublished 2024-05-30, reviewed 2026-07-13original-paper, published-third-party |
Claims and evidence
Claim ledger, 2OSWorld provides 369 open-ended computer tasks in real Ubuntu, Windows, and macOS environments across web, office, media, coding, and cross-application workflows.
The listed result anchors are third-party publications tied to explicit evaluation settings and are not evaluations run by Draconitic.