Description
Windows Agent Arena (WAA) is a novel, open-source framework designed to address the challenges in evaluating and developing AI agents capable of reasoning, planning, and acting within a real Windows operating system. Traditional benchmarks often focus on limited modalities or domains and are time-consuming, taking days for full evaluations due to the sequential nature of tasks. WAA offers a reproducible and realistic Windows OS environment where AI agents can interact with a wide range of applications, tools, and web browsers, mirroring human user experiences.
Read More
The framework supports the deployment of agents at scale, leveraging Azure ML cloud infrastructure for parallel execution. This allows for hundreds of tasks to be benchmarked in minutes rather than days. WAA includes over 154 diverse tasks spanning common Windows workloads such as document editing (LibreOffice Calc/Writer), internet browsing (Microsoft Edge, Google Chrome), system tasks (File Explorer, Settings), coding (Visual Studio Code), media playback (VLC Player), and utility functions (Notepad, Clock, Paint). Task evaluation is deterministic, with custom scripts generating rewards at the end of each episode.
At its core, WAA utilizes a Docker container hosting a Windows 11 VM. A Python Flask server acts as an intermediary, executing commands within the VM and returning observations. For scalable cloud parallelization, Azure Machine Learning jobs are employed, distributing tasks evenly among compute instances and aggregating results. This infrastructure supports flexible local execution during prototyping and robust cloud-based benchmarking.
To demonstrate WAA's capabilities, the Navi agent was introduced. Navi uses chain-of-thought prompting to reason about the computer's state, past actions, and decide on the next step. Its input includes the foreground window title, titles of all open windows/tabs, and a screen representation processed through various methods like UIA tree parsing, DOM tree parsing, OCR, icon/image detection, and OmniParser. Initial results show that while current models achieve lower performance than humans, the quality of screen representation significantly impacts agent success, with UIA tree parsing and OmniParser's icon captioning showing performance boosts.
WAA is a valuable resource for researchers and developers aiming to advance the field of AI agents, particularly those operating in complex desktop environments. It facilitates rigorous testing, comparative analysis of agent architectures, and the generation of insights for future agent development.
- CategoryAI Agent Builders
- Pricing ModelOpen source
- DeploymentCloud (Azure ML), Local
- PlatformsWindows
Windows Agent Arena's Core Features
Scalable Windows OS environment for AI agents
Realistic desktop interaction with diverse applications
Supports multi-modal AI agents
Benchmarking of agentic workflows
Parallel execution via Azure ML
154 diverse Windows tasks
Deterministic task evaluation
Chain-of-thought prompting for agents
Multiple screen representation methods (UIA, DOM, OCR, Icon Detection, OmniParser)
Open-source framework
Reproducible research environment
Getting Started with Windows Agent Arena
Set up the Docker container with Windows 11 VM
Configure task scheduling and agent deployment
Define agent inputs and screen parsing methods
Run benchmark evaluations locally or on Azure ML
Analyze agent performance metrics and logs
Windows Agent Arena's Use Cases
- AI Agent Benchmarking
- Desktop Automation Research
- Multi-modal Agent Development
- Reproducible AI Research
- Human-AI Interaction Studies
