Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It also depends on your skills and tooling (test execution and verification- agent browser, functional/unit etc) GPT 5.6 Sol lagging behind Kimi, GLM 5.3 is surprising to me.
IMO Fable 5.1 ~ Astra > GPT 5.6 Sol > Opus.
- this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.by visiondude
- I used a similar methodology. Code review is my most requested action, so I used blind code review results to compare the frontier AIs.
Even posted an article about it:
https://www.matheusmoreira.com/articles/code-reviewing-lone-...
Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced some interesting results.
Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.
- I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.by jstummbillig
- My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
by lmeyerov - I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there.
I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things.
I've tried reversing this pairing with Fable as advisor. It'll just sit there going "sounds good"
by glub - I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.
- Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.