Description
Portal2Bench is a specialized benchmark designed to assess the problem-solving capabilities of AI agents within the intricate environment of Valve's Portal 2. The platform measures how effectively AI agents can navigate and complete a series of 26 distinct test chambers, drawn from chapters 2 to 5 of the game. Each chamber represents a fresh, independent challenge for the agent.
Read More
The core methodology involves pausing Portal 2 while the AI model processes information and formulates its next actions. The agent interacts with the game by sending key presses and view turns, receiving real-time feedback in the form of screenshots and its current position within the game world. To ensure fair and controlled evaluation, agents are provided with a limited 'notes folder' for internal processing but are strictly denied access to external resources like a shell, web, or console.
A chamber is considered 'solved' only if the AI agent successfully reaches the next map within stringent limits: a maximum of 300 'plans' (presumably actions or computational steps) and a cost ceiling of $30, calculated based on API list prices for model usage. This dual constraint emphasizes both efficiency and economic viability in AI agent performance.
Portal2Bench also addresses concerns about 'cheating' by implementing robust QA checks. Agents are confined to their designated notes folder, preventing any unauthorized external interaction. In the event of an agent's in-game 'death,' it respawns at the last checkpoint, mirroring normal Portal 2 gameplay, and is simply informed of its demise without additional context. The benchmark explicitly states it is not affiliated with or endorsed by Valve, ensuring its independent research focus.
- CategoryAI Agent Builders
- Pricing ModelAPI List Prices
- DeploymentCloud
- PlatformsWeb
Portal2Bench's Core Features
AI Agent Performance Benchmarking for Portal 2
Evaluation across 26 distinct test chambers
Strict limits on plans (300) and cost ($30) per chamber
Real-time game interaction via key presses and view turns
Visual input through screenshots and positional data
Isolated environment with no shell, web, or console access
Internal 'notes folder' for agent processing
Respawn mechanism at checkpoints upon agent death
API list pricing used for cost calculation
QA checks to prevent unauthorized agent actions
Leaderboard tracking for various AI models
Independent research platform, not affiliated with Valve
How to use Portal2Bench?
Understand the Goal: Learn that Portal2Bench evaluates AI agents' ability to solve Portal 2 test chambers.
Review Benchmarking Criteria: Note that success is defined by solving a chamber within 300 plans and a $30 cost limit.
Observe Agent Interaction: See how agents send keys and view turns while receiving screenshots and position data.
Check Model Performance: View the leaderboard to compare how different AI models perform on the benchmark.
Analyze Cost and Efficiency: Examine the cost per chamber, plans per chamber, and time per chamber metrics for insights.
Portal2Bench's Use Cases
- AI Agent Performance Evaluation
- Game AI Research
- Comparative AI Benchmarking
- Cost-Efficiency Analysis of AI Models
- AI Planning and Strategy Development






