OSWorld history

Task and data. OSWorld provides 369 open-ended computer tasks in real Ubuntu, Windows, and macOS environments across web, office, media, coding, and cross-application workflows.

Scoring. Execution-based success rate is the share of tasks whose environment end state satisfies the task-specific evaluator; higher is better under the same OS and observation/action setup.

What it measures. multimodal computer-agent ability to perceive interfaces and execute mouse, keyboard, and code actions across real desktop applications.

Comparison rules. Preserve input modality, action space, operating-system image, model snapshot, and task release. Human, accessibility-tree, screenshot, and Set-of-Mark rows are distinct.

Access and licensing. The Compendium does not reproduce tasks, screenshots, application assets, or environment images; use official access paths and third-party software licenses.

Task and data. OSWorld provides 369 open-ended computer tasks in real Ubuntu, Windows, and macOS environments across web, office, media, coding, and cross-application workflows.

Scoring. Execution-based success rate is the share of tasks whose environment end state satisfies the task-specific evaluator; higher is better under the same OS and observation/action setup.

What it measures. multimodal computer-agent ability to perceive interfaces and execute mouse, keyboard, and code actions across real desktop applications.

Comparison rules. Preserve input modality, action space, operating-system image, model snapshot, and task release. Human, accessibility-tree, screenshot, and Set-of-Mark rows are distinct.

Access and licensing. The Compendium does not reproduce tasks, screenshots, application assets, or environment images; use official access paths and third-party software licenses.

Task and data. OSWorld provides 369 open-ended computer tasks in real Ubuntu, Windows, and macOS environments across web, office, media, coding, and cross-application workflows.

Scoring. Execution-based success rate is the share of tasks whose environment end state satisfies the task-specific evaluator; higher is better under the same OS and observation/action setup.

What it measures. multimodal computer-agent ability to perceive interfaces and execute mouse, keyboard, and code actions across real desktop applications.

What it does not measure. safe deployment, reliability on every OS version, accessibility, cost, latency, recovery from destructive actions, or productivity in a user's actual environment.

Limitations. Environment versions drift, some software activation is restricted, agent histories and observation modes differ, and execution evaluators do not capture every qualitative failure.

Comparison rules. Preserve input modality, action space, operating-system image, model snapshot, and task release. Human, accessibility-tree, screenshot, and Set-of-Mark rows are distinct.

Access and licensing. The Compendium does not reproduce tasks, screenshots, application assets, or environment images; use official access paths and third-party software licenses.

Result-reading note. Every result below was published by the named third party and reviewed by the Compendium on 2026-07-13. Flintglade did not run, reproduce, certify, or independently validate these evaluations. A score only belongs to its recorded benchmark version, split, prompt or scaffold, metric, and subject version; omitted setup details are called out instead of guessed.