Description
StarSkirmish Bench v0.2 is a unique benchmarking platform designed to evaluate the capabilities of large language models (LLMs) in writing and optimizing Protoss bots for the iconic game StarCraft: Brood War. The platform challenges each LLM to write a bot in C++ within a one-hour timeframe. These bots are then tested against a roster of competitive human-written bots and other LLM-generated bots to assess their performance.
Read More
The evaluation focuses on long-horizon reasoning and agentic coding, as the LLMs must learn from practice games to improve their bots. The platform features flagship models from leading AI labs such as OpenAI, Anthropic, Google, and Meta, each running in its lab's first-party CLI harness. The performance of each LLM is measured by its average win rate across five runs against a fixed roster of bots.
StarSkirmish Bench v0.2 introduces several updates from its predecessor, including new models like Gemini 4 Argon and Muse Wonder Full. The top-performing model, GPT-6 Astra, demonstrates a significant lead over others, although human-written bots still outperform all LLMs in the most challenging matches.
The platform also provides insights into the cost-effectiveness of each model, with GPT-6.1 Sol offering a compelling balance of performance and cost. Despite the advancements, the strongest human-written bots remain ahead, highlighting the ongoing challenge of developing AI capable of matching human ingenuity in complex strategic games.
- CategoryAI Game Development Tools
- Pricing ModelNot specified
- DeploymentCLI
- PlatformsOpenBW
StarSkirmish Bench v0.2's Core Features
Benchmarking platform for LLMs
Protoss bot development in C++
Evaluation against human-written bots
One-hour development timeframe
Performance measured by win rate
Includes flagship models from top labs
Cost-effectiveness analysis
Focus on long-horizon reasoning
How to use StarSkirmish Bench v0.2?
Configure: Set up the CLI harness
Develop: Write a Protoss bot in C++
Test: Run practice games against bots
Evaluate: Measure performance by win rate
StarSkirmish Bench v0.2's Use Cases
- AI Model Evaluation
- Bot Development
- Cost-Effectiveness Analysis
- Long-Horizon Reasoning
- Agentic Coding
