Discussion summary
Discussions focus on the relevance of benchmarks in evaluating AI models, with some arguing that real-world performance should be the ultimate test. Participants mention recent model releases, personal testing experiences, and the trend of creating private benchmarks.
What the discussion says
- Some believe benchmarks are becoming less relevant if models are close to AGI.
- Others suggest building private benchmarks tailored to specific needs.
- Concerns about models' limitations in spatial reasoning and practical tasks.
- Emphasis on creating scaffolds and constraints rather than relying solely on raw model capabilities.
“If we're close to AGI, proof should be in the pudding.”
“Models don't perform well on functional parts from descriptions.”
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a pretty solid path to take.by jheitmann
- All of the benchmarks are pretty terrible when you look under the hood.
For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1
At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.
The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.
A few examples:
- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.
- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].
- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].
These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).
The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.
[0] https://github.com/harbor-framework/terminal-bench-2-1/issue...
[1] https://github.com/harbor-framework/terminal-bench-2-1/issue...
by jumploops - Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.by shay_ker
- Seems like depending on your field these days, the hot thing to do is build your own private benchmarks.
In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts.
They just don't understand PVC parts, triggers, etc.
by bellowsgulch - Based on the numbers here it seems there’s less than 800 tasks in the entire benchmark. That is enough for a handful of engineers to comb through in a week (which is what OpenAI eventually did here).
On the one hand, kudos to them for actually doing that work.
On the other hand, garbage in, garbage out. It’s a bit embarrassing for the original authors to have not actually checked, and it’s embarrassing for everyone downstream to have not checked either.
Also if you check the article, although an LLM did find issues, it tended to underestimate issues that professional software engineers found.
by janalsncm - Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.by mlhpdx
- There are also a lot of fake results out there on Terminal Bench 2 for different reasons (although the great team behind it Ryan/Alex et al, recently cleaned up a lot of dodgy submissions). A lot of labs publish the results by modifying timeouts or hardware config which effectively bypasses what is being tested in certain tasks. Then there is harness level cheating, models reward hacking and more...
In fact, one thing that still bothers me after months is the gpt-5.5 official submission. This task in particular https://www.tbench.ai/leaderboard/terminal-bench/2.0/codex/0...
The task has the following timeouts (https://github.com/harbor-framework/terminal-bench-2/blob/ma...).
[verifier]
timeout_sec = 1200.0
[agent]
timeout_sec = 1200.0
[environment]
build_timeout_sec = 600.0
Which means no agent should take more than 3000 seconds doing it. Two out of five attempts in the link above took well over 3000 seconds (75min and 80 min respectively). Even though they failed, the fact that they ran that long is sus.
Goodhart’s Law at work
- I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests?
Give us something that measures a combination of efficiency and intelligence.
I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have budget to self-test.
by jjcm