Discussion summary
Discussions focus on the relevance of benchmarks in evaluating AI models, with some arguing that real-world performance should be the ultimate test. Participants mention recent model releases, personal testing experiences, and the trend of creating private benchmarks.
What the discussion says
- Some believe benchmarks are becoming less relevant if models are close to AGI.
- Others suggest building private benchmarks tailored to specific needs.
- Concerns about models' limitations in spatial reasoning and practical tasks.
- Emphasis on creating scaffolds and constraints rather than relying solely on raw model capabilities.
“If we're close to AGI, proof should be in the pudding.”
“Models don't perform well on functional parts from descriptions.”
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.by xacky
- This ties into the bias-variance tradeoff (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) common with building non-LLM models. The solutions can only be a) figure out how to get LLMs smaller with similar performance so they don't memorize things/game the benchmarks and b) build benchmarks that are indeed comprehensive for all real-world data, which is infeasible.by minimaxir
- AGI is a long way off. Unless you’re talking about some unknown-to-me LLM marketing BS which is called “AGI” or something, I guess. Artificial general purpose intelligence is so different to LLMs or image AI that they are completely incomparable, except to say that they are all artificial. AGI will do a lot more than token prediction.by naikrovek
- they should be consulting Donald Rumsfeld and make sure they implement the Unknown-Unknowns benchmark, because thats how they get youby cyanydeez
- Unless they have something in the labs that massively departs from their current products, AGI isn't on the table and is purely hype for marketing purposes.by metalliqaz
- Lately my benchmark is build123d - trying to force them to build me functional parts only by the description. All of the models don't perform well.by ReptileMan
- IDK, sounds like it has brute forced my password already.by mgiampapa
- was watching some youtube about the text/vision models and the trouble with language as a descriptor; their novel idea was to use coordinate systems on visual imput so the model would map out an image, then it tags what's looking at with coordinates or boxes, and then think with those box coordinates, providing a level of disambiguation.
If you tried to explain the same stuff using purely text, you'd also need to come up with some kind of "language", which you know is programming.
So 1: you'd need to fine tune a model to actuall succeed; they're not reaching AGI any time soon with the current batch. 2: you need to develop a lingual DSL for it, as they'll never do much of anything without some kind of glue and disambiguation.
by cyanydeez - I can understand why this might make for a bad benchmark, but
Overly strict tests enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.
Underspecified prompts omit requirements that hidden tests enforce and that are not reasonably inferable.
Low-coverage tests under check the requested feature, so incomplete fixes can pass.
A misleading prompt points models toward the wrong behavior or contradicts what tests require.
If the goal is, "how does my model compare to real SWEs", these are pretty reasonable situations that your model will have to encounter. It's a little like making a nursing exam and then flagging that some of the tests required you to ask the attending doctor for additional information that's not in the chart, or that the patient's family didn't fully explain their aging grandma's medical history.
I can understand why they might want a tighter benchmark, but if you're OpenAI and you promised your model as a replacement for real workers, this isn't the best look. It seems like you would want to test these things.
by jimberlage - What is considered SOTA for SWE benchmarks now?by dandaka
- strawberryby carabiner
- FrontierBench
- https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)by swyx
- I've generally found DeepSWE[0] to be pretty true to reality.by EuanReid
- Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.by Topfi
- It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a pretty solid path to take.by jheitmann
- If they fixed it, then it wouldn't be SWE-Bench Pro anymore, right? It'd be "SWE-Bench-Pro-Fixed-OpenAI." I think it's better optics for the independence of the benchmark if the OpenAI team lets some third party do the fixing and release the improved benchmark.
...Although OpenAI did exactly that when they released SWE-Bench Verified, so maybe I'm talking out of my butt here.
by Centigonal - Pointing out problems (e.g., hidden tests that assume narrow implementation details) is much easier than fixing them (e.g., creating tests that work for any possible choice of implementation).by tedsanders
- And it reads to me like they have some other reason to move on from SWE Bench Pro, but they don't want to say what it is. They say right up top, "~30% of the tasks are broken." But that leaves ~70% un-broken, which seems pretty good to me. It would be nice if they would also say: "Here's the list of instances that are broken: <CSV>". Or, "Here's the subset of SWE Bench Pro we will use going forward." They're letting the perfect be the enemy of the good.by ebcode
- All of the benchmarks are pretty terrible when you look under the hood.
For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1
At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.
The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.
A few examples:
- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.
- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].
- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].
These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).
The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.
[0] https://github.com/harbor-framework/terminal-bench-2-1/issue...
[1] https://github.com/harbor-framework/terminal-bench-2-1/issue...
by jumploops - Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.by shay_ker
- SWE-Bench Pro was created to replace SWE-Bench and fix these problems.by paxys
- Seems like depending on your field these days, the hot thing to do is build your own private benchmarks.
In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts.
They just don't understand PVC parts, triggers, etc.
by bellowsgulch - Or you’re getting steered into la la land because of your prompt
- Or defensively expect models to be stupid.
Seems the smart thing to do is not assume an agent will do the right thing. But to create the scaffold / harness that enforces constraints to steer them towards a good result.
Then you can swap out the really smart model for maybe something cheaper.
by softwaredoug - It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever.
What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.
Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.
The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.
by ACCount37 - Based on the numbers here it seems there’s less than 800 tasks in the entire benchmark. That is enough for a handful of engineers to comb through in a week (which is what OpenAI eventually did here).
On the one hand, kudos to them for actually doing that work.
On the other hand, garbage in, garbage out. It’s a bit embarrassing for the original authors to have not actually checked, and it’s embarrassing for everyone downstream to have not checked either.
Also if you check the article, although an LLM did find issues, it tended to underestimate issues that professional software engineers found.
by janalsncm - Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.by mlhpdx
- This was also my thought, and I think holds true of the ones with invisible requirements that aren't stated up front and are only captured in tests. Oh, you need to rework your solution to handle requirements nobody mentioned before? Well, me too.
Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.
(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)
by mkozlows - The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions.
So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust verifier and run it cheaply.
by kakugawa