Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Slop
> We find that nearly half of the our bench- marks exhibit saturation
by buckle8017 - It's a grammatical error, sure, where is the indication of slop?by jdiff
- This leads me to believe it's NOT slop lolby joeyagreco
- This is AI slop. They didn't even change the plots default template.by behnamoh
- Check out https://agents-last-exam.org/ there is still room for improvements!by hagen8
- So commonsenseqa is the least saturated benchmark? It's like authors don't even read what they publish.
- 37 authors and contributors need to be named up top?
Oh! I got my name on a paper! I don't think there is much reward for it these days.
by kanbankaren - That's a lot of words to say you need larger set of questions for today's models. 300 questions won't be enough to find the differenceby mohsen1
- It appears y combinator has removed this interesting paper from its top trending position. Gee I wonder why?by otterdude
- only 30 upvotes in 3 hours but 50 comments(many negative) is not a great signal.by JacobAsmuth
- I think its been pretty clear that in abnsense of clear use cases that are monetizable many model providers have been benchmaxxing on abstract or low utility average user performance.
This results in a lot of "oh wow it can do math I dont care about" and "it can't code a lot, but not well" outcomes instead of the core needs:
1) Cheaper faster and real time 2) Long walk capable without losing attention while rescoring goals over updated enviroment 3) Specific domain knowledge that can be trained quickly into the model (how we do work in this specific case)
by tsunamifury - As a PC gamer who grew up in the 00s, this has been something I’ve tried to warn ardent LLM and model enthusiasts about for quite some time.
Benchmarks are handy when they’re new, novel, and constantly changing. The second you let even a single aspect of it stagnate, it becomes a gameable score rather than a useful metric. In PC Gaming, we saw vendors optimize for specific titles, benchmark tools, and scenarios at the expense of general performance, and eventually the industry had a “come to Jesus” moment where we had to collectively decide how to move forward from an industry built on thoroughly gamed benchmarks, with entities like Gamers’ Nexus and Digital Foundry being the end results of that falling out.
LLMs were always going to end up the same way, because the people building the benchmarks - well-intentioned as they were - ultimately fell into the exact same traps with fixed scoring rubrics, known test questions, and believing in some form of “completeness” that could be attained or achieved. The net result are models consistently scoring better on benchmarks but also seeing diminishing returns and rising vulnerabilities, because actual improvement or utility isn’t what they’re being optimized for so much as bragging rights. It’s why there’s so much growing interest in things like MoE execution on unified memory platforms as a means of porting larger models to consumer kit, or ternary models (shoutout to Bonsai) as a means of reducing overall size: both take leading edge, benchmark-saturating models and show that with minimal score loss, they function about as well as frontier models might.
Building a new benchmark won’t solve the problem, either. To move forward, we must evaluate LLMs objectively and with continuously evolving workloads. More “pelican on a bicycle” stuff, but from varying perspectives and use cases. Radiologists putting models through their paces with usable sample data they don’t share with AI labs, or IT folks tasking agents with bootstrapping specific, real-world workloads. To prove general intelligence, we need more specialists evaluating them specifically and generally in ways that are transparent to consumers but difficult or impossible for AI companies to prepare against.
Only then will scoring values matter.
by stego-tech - > Building a new benchmark won’t solve the problem, either.
It will if the benchmark is proprietary. If you can't train on it, then it's extremely difficult to game, and if it's hard enough, then it's economically more efficient to just...make the model smarter
by throw10920 - I started thinking about this back after the Llama 4 release, and since then our team has put a lot of thought into designing evaluations that don't saturate, are resistant to contamination, and can scale. What has worked best for us is using multi-agent environments with open-ended cooperative or competitive goals. Mostly designed as multiplayer games. The results tend to align with our experience for coding better than any non-aggregator benchmark, and likely at lower cost to run.
Data at https://gertlabs.com/rankings
by gertlabs - If you're ranking Opus > Fable you're ranking "do [clearly defined thing with easy to grade endpoint]" too much. Real world doesn't value that nearly as much and it's why benchmarks are maxxed.by nwienert
- I've developed a benchmark that I think should be resistant to saturation, is easily verifiable, and anecdotally correlates with desirable behavior (ability to not get confused while generating text with state).
I think it's interesting, I think other people would find it useful, but I don't want to spend a bunch of money running it against all the frontier models.
What's the best way to reach out to labs like yours to collaborate on something like that? Are there any labs that are more open to submissions from internet randos?
by erikwiffin - You post your benchmark on every other AI article, I've seen you do this by now more than a dozen times. It's a bit much. I don't want to be too harsh but your benchmark is obviously flawed when the top 3 models for Typescript (Combined) are Grok 4.5, Muse Spark 1.1 (lol), Gemini 3.5! Flash and then followed by Luna, beating Opus 5, Fable, 5.6 Sol etc by quite some margin. In fact 5.6 Sol ranks lower than Kimi K2.7 Code and even Grok Build 0.1. There are so many entries in your rankings that don't make any sense whatsoever that I can't take this benchmark serious and I have not seen it gaining traction. Please stop spamming it?by eis
- This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression.
If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence.
I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm
“The seeker after truth is not one who studies the writings of the ancients and, following his natural disposition, puts his trust in them, but rather the one who suspects his faith in them and questions what he gathers from them, the one who submits to argument and demonstration and not the sayings of human beings whose nature is fraught with all kinds of imperfection and deficiency. Thus the duty of the man who investigates the writings of scientists, if learning the truth is his goal, is to make himself an enemy of all that he reads, and, applying his mind to the core and margins of of its content, attack it from every side. he should also suspect himself as he performs his critical examination of it, so that he may avoid falling into either prejudice or leniency.” - ibn al-Haytham
by otterdude - Assuming you are completely correct about the 80/20 rule, we have evidently not yet reached that 80%. Who can say when it will be achieved? The ceiling is glass, we have to touch it to know where it is.by grim_io
- Even without getting better trained models and only speed increase, the output would be dramatically better. An LLM or non llms that is a billion times faster than now would be so insanely strong in many areas.by holoduke
- >This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression.
It's just inadequate benchmarks. Anyone who has used Fable for anything particularly difficult will have seen that it's miles ahead of Opus 5.0, yet the majority of benchmarks are completely unable to capture this.
by logicchains - It's also important not to put too much faith into ancient sayings and aphorisms.
As a civilization, we are currently brushing up against the physics of efficiency. In many areas we have achieved close to what is theoretically possible, based on physics.
Such was not the case for the majority of human existence.
The body of research a.k.a. "writings of the ancients" is now insurmountably higher than it would have been during the time of ibn al-Haytham, when any kind of writing at all was scarce and literacy was low.
by antisthenes - I think LLMs will continue to improve in the capabilities which they are demonstrably good at, but there are many things which they are not good at which it is not cost effective or meaningful to improve, and in these areas we will not consider them “intelligent,” in the same way that we don’t consider computers “intelligent” but do find them very good at doing wrote calculations.by DiscourseFan
- Idk about end of the road, I’m sure they can squeeze out some more performance by curating even more data and doing even more RL.
But I would bet that pretty much all of the improvement we’ve seen over the last year with coding has come from RL, not from the models becoming particularly stronger. And this makes sense, if models grow sublinearly with compute. And it seems like they do.
by slopinthebag - I really don’t think we know enough about what „intelligence“ is or how LLMs actually work to confidently say that this is the end of the road for LLM.by adrianN
- >This seems to be the end of the road for LLM's
This is an amazingly ignorant thing to say given the current pace of progress.
by heaney-555