Description
WebVoyager is an innovative Large Multimodal Model (LMM) powered web agent designed to complete user instructions end-to-end by interacting with real-world websites. This project provides the code and data for the WebVoyager paper, showcasing its capabilities in multimodal web navigation and task completion.
Read More
The core of WebVoyager lies in its ability to integrate both textual and visual information, allowing it to understand and interact with web pages more effectively. It employs a generalist planning approach for navigation, enabling it to tackle a diverse range of web tasks. The system is built upon an online web browsing environment, which is implemented using Selenium, providing a realistic platform for testing and development.
To facilitate research and development, WebVoyager includes a curated dataset of task queries. This dataset was generated using a semi-automated approach, resulting in 643 task queries spread across 15 websites, with each website featuring over 40 queries. The data is available in `data/WebVoyager_data.jsonl`, and reference answers are provided in `data/reference_answer.json`. Additionally, the project incorporates 90 web browsing tasks extracted from the GAIA dataset, available in `data/GAIA_web.jsonl`.
For users looking to expand the task pool, the project suggests using GPT-4 to generate new tasks. Instructions and example prompts are provided to guide this process, emphasizing diversity and solvability within specific websites. The setup involves creating a conda environment, installing dependencies via `requirements.txt`, and configuring the `run.sh` script with an OpenAI API key.
WebVoyager offers flexibility in its execution, supporting both standard and text-only modes, with adjustable parameters for maximum iterations, attached images, and model temperature. The project also includes an automated evaluation protocol using GPT-4V, detailed in the `evaluation` directory, to assess the agent's performance on completed tasks. The results of the agent's interactions, including messages and screenshots, are saved for human review and analysis.
- CategoryBrowser & Web Automation
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
WebVoyager's Core Features
End-to-end web agent powered by Large Multimodal Models (LMMs)
Integrates textual and visual information for web task completion
Generalist planning approach for web navigation
Online web browsing environment built with Selenium
Dataset of 643 diverse web task queries across 15 websites
Tasks extracted from the GAIA dataset
Supports task expansion using GPT-4
Configurable execution parameters (max iterations, temperature, headless mode)
Includes an automated evaluation protocol using GPT-4V
Saves interaction logs and screenshots for analysis
Getting Started with WebVoyager
Setup Environment: Create a conda environment, activate it, and install dependencies using `pip install -r requirements.txt`.
Prepare Tasks: Copy desired test tasks into `data/tasks_test.jsonl`. Manually update dates for time-sensitive tasks.
Configure API Key: Modify the `api_key` in the `run.sh` script with your OpenAI API key.
Execute WebVoyager: Run the agent using the command `bash run.sh`.
Monitor Results: Check the `test_tasks.log` file for execution output and the `output_dir` for detailed interaction logs and screenshots.
WebVoyager's Use Cases
- Web Task Automation
- Web Navigation Research
- Data Collection
- AI Agent Development
- Website Interaction Testing
