Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.by BeaverGoose
- It’s been interesting to watch Artosis (big StarCraft streamer) go silent on Pluto and other bots in the last few weeks. At the start, he was interested and showed a few games where all the pros lost except when they exploited glitches in the AI. Then he lost several times against it and he stopped casting AI games.
Smells like fear
by hitekker - >a benchmark where LLMs play StarCraft BW by writing code
Around the time Starcraft 2 came out, I talked about it with a friend who was very enthusiastic about the subject. He said he thought the game was awesome, but it was really too bad that it imposed all this mindless tedium on you. (He was a programmer, of course!)
He said it would be really cool if you could just give higher level directives to your units. As though you were commanding them from far away. (The game, in his vision, would somehow account for speed of light, across vast distances.)
I think he really just wanted to play Civ or something :) But this post reminded me of that conversation.
by andai - So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966by MichaelNolan
- > ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
> The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.Anthropic (Claude models): Claude Code OpenAI (GPT models): Codex Google (Gemini models): Antigravity CLI Meta (Muse models): Muse CodeOne thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
3. thinking levels
by djoldman - StarCraft probably has hundreds of bot implementations lying on GitHub. Which makes this bench a recall-from-the-training-set competition, essentially. I tried that method on old games that certainly do not have a ton of bot implementations on GitHub. It revealed some interesting aspects about LLM reasoning and ways to use it. https://replicated.live/blog/games
Overall, this is a useful benchmarking approach. May take some newer papers and evaluate them this way.
by gritzko - Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
by cgearhart - Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.