AI
StarSkirmish benchmark finds leading models still struggle against strong StarCraft bots
The StarSkirmish Bench v0.2 gives language models one hour to build Protoss bots for StarCraft: Brood War, then measures them against fixed human- and demo-bot rosters. GPT-6 Astra leads, while GPT-6.1 Sol offers the strongest cost-adjusted result; Gemini 4 Argon ranks third but shows unusually high run-to-run variation. Even the best model wins only 7% against the strongest human bots, suggesting substantial limits in long-horizon agentic coding.
Editorial summary · Publisher article
What matters
- Scores average five runs, with each bot evaluated across 13 opponents, three maps and fresh seeds.
- Models that practiced earlier against strong opponents generally scored better, though Claude Fable 5.1 was an exception.
- The benchmark’s results correlate weakly with some widely reported evaluations, highlighting task-specific capability differences.
- The v0.2 comparison uses each lab’s first-party CLI harness, so harness differences remain part of the comparison.
Context & caveats
- These results measure performance on one constrained StarCraft task and should not be treated as a general ranking of model capability.
Why this made the edition
The benchmark provides unusually detailed methodology, comparative results, behavioral traces and limitations, making it useful evidence about long-horizon coding performance despite being a single evaluation.
Source material