Website• Paper• Doc• Data• Data Viewer• Discord

📢 Updates

• 2024-06-15: We refactor the code of environment part to decompose VMware Integration, and start to support other platforms such as VitualBox, AWS, Azure, etc. Hold tight!
• 2024-04-11: We released ourpaper,environment and benchmark,andproject page.Check it out!

💾 Installation

On Your Desktop or Server (Non-Virtualized Platform)

Suppose you are operating on a system that has not been virtualized, meaning you are not utilizing a virtualized environment like AWS, Azure, or k8s. If this is the case, proceed with the instructions below. However, if you are on a virtualized platform, please refer to thevirtualized platformsection.

1. First, clone this repository andcdinto it. Then, install the dependencies listed inrequirements.txt.It is recommended that you use the latest version of Conda to manage the environment, but you can also choose to manually install the dependencies. Please ensure that the version of Python is >= 3.9.

#Clone the OSWorld repository
git clone https://github.com/xlang-ai/OSWorld

#Change directory into the cloned repository
cdOSWorld

#Optional: Create a Conda environment for OSWorld
#conda create -n osworld python=3.9
#conda activate osworld

#Install required dependencies
pip install -r requirements.txt

Alternatively, you can install the environment without any benchmark tasks:

pip install desktop-env

2. InstallVMware Workstation Pro(for systems with Apple Chips, you should installVMware Fusion) and configure thevmruncommand. The installation process can refer toHow to install VMware Worksation Pro.Verify the successful installation by running the following:

vmrun -T ws list

If the installation along with the environment variable set is successful, you will see the message showing the current running virtual machines.

Note:We also support usingVirtualBoxif you have issues with VMware Pro. However, features such as parallelism and macOS on Apple chips might not be well-supported.

All set! Our setup script will automatically download the necessary virtual machines and configure the environment for you.

On AWS or Azure (Virtualized platform)

On your AWS

SeeAWS_GUIDELINEfor using AWS as the virtualized platform. Please carefully go through the guideline and choose the proper instance type and region.

On your Azure

We have finished the support for Azure but not yet fully tested.

Others

We are working on supporting more 👷. Please hold tight!

🚀 Quick Start

Run the following minimal example to interact with the environment:

fromdesktop_env.desktop_envimportDesktopEnv

example={
"id":"94d95f96-9699-4208-98ba-3c3119edf9c2",
"instruction":"I want to install Spotify on my current system. Could you please help me?",
"config":[
{
"type":"execute",
"parameters":{
"command":[
"python",
"-c",
"import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"
]
}
}
],
"evaluator":{
"func":"check_include_exclude",
"result":{
"type":"vm_command_line",
"command":"which spotify"
},
"expected":{
"type":"rule",
"rules":{
"include":["spotify"],
"exclude":["not found"]
}
}
}
}

env=DesktopEnv(action_space="pyautogui")

obs=env.reset(task_config=example)
obs,reward,done,info=env.step("pyautogui.rightClick()")

You will see all the logs of the system running normally, including the successful creation of the environment, completion of setup, and successful execution of actions. In the end, you will observe a successful right-click on the screen, which means you are ready to go.

🧪 Experiments

Agent Baselines

If you wish to run the baseline agent used in our paper, you can execute the following command as an example under the GPT-4V pure-screenshot setting:

SetOPENAI_API_KEYenvironment variable with your API key

exportOPENAI_API_KEY='changme'

python run.py --path_to_vm Ubuntu/Ubuntu.vmx --headless --observation_type screenshot --model gpt-4-vision-preview --result_dir./results

The results, which include screenshots, actions, and video recordings of the agent's task completion, will be saved in the./resultsdirectory in this case. You can then run the following command to obtain the result:

python show_result.py

Evaluation

Please start by reading through theagent interfaceand theenvironment interface. Correctly implement the agent interface and import your customized version in therun.pyfile. Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent.

❓ FAQ

What is the username and password for the virtual machines?

The username and password for the virtual machines are as follows:

• Ubuntu:user/password

How to setup the account and credentials for Google and Google Drive?

SeeAccount Guideline.

How can I configure a proxy for the VM if I'm behind a GFW?

SeeProxy Guideline.

What are the running times and costs under different settings?

Setting	Expected Time*	Budget Cost (Full Test Set/Small Test Set)
GPT-4V (screenshot)	10h	$100 ($10)
Gemini-ProV (screenshot)	15h	$0 ($0)
Claude-3 Opus (screenshot)	15h	$150 ($15)
GPT-4V (a11y tree, SoM, etc.)	30h	$500 ($50)

*No environment parallelism. Calculated in April 2024.

📄 Citation

If you find this environment useful, please consider citing our work:

@misc{OSWorld,
title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments},
author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},
year={2024},
eprint={2404.07972},
archivePrefix={arXiv},
primaryClass={cs.AI}
}