Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Auto model selection has been basically unusable for me, so I'm really unsure how much to trust the post's narration on this...
Oh well, can't wait to be harassed by management to try and use it anyways...
by perching_aix - This looks very similar to what I've seen with a different harness - https://github.com/AMAP-ML/LongHorizon-Harness. It's critique pattern looks like another version of lh-harness without looping toward an overarching goal
- This has got to be the worst project name I've seen all year.by zatkin
- They should probably focus on “project hydrastable infrastructure” insteadby fnoef
- i've been using adversarial critique and reviews for many planning, solution design and implementation steps inside workflows.
It's so effective and helps catching so many design flaws, implementations misses etc ... that i'm wondering how people manage to build complex/large projects with agents without this kind of process. Well, i actually built this thing because i couldn't get good results so i had to find a way.
I'm gonna open source the whole thing but it needs some cleanup, there's a basic landing page here https://kodfactory.com if anyone wants to be notified when it's released on github. Yeah i know, the world really needs another software factory :-)
by guybedo - I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness.
This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair.
Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics: - tweaks few decode settings like slightly higher temperature - detects when model gets stuck and tells it to "go on" - detects responses cut in the middle, empty responses that contain only reasoning, formats not passing verification etc and tells the model to "try better"
And that is it. 10% more. I admit on a subset of tasks, but results are results, even on a subset.
by Roark66 - If you’re a copilot cli user (we exist) you know that the auto mode leaves something to be desired. It only changes models at the start of a session or after compaction. Most dev tasks are now sub agent heavy and there is no model routing on the sub agents.
I do notice that there is no mention of effort levels in the comparisons. 5.6 Luna Max is really good and really cheap. A 5.6 Sol high orchestrator with 5.6 Luna max is cheap, has frontier level performance, and is faster than Sol alone. This can be accomplished with simple agent instructions. Looking forward to running my own benchmarks on HydraFusion to see how it fares. Gone are the days of a single model doing all of the work it seems, unless the work requires no tool calls.
by hdz - > One model drafts a result, an independent read-only critic from a different model family reviews it
Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does.
Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor.
The ablations [1] proved that neither company alone was better than using both.
The paper was a general response to "What does your company do that Anthropic can't?" but more so a demonstration of how to make something 90%+ good with models which eval at 60% or so (& Gas Town post unblocked our "this is a trade secret" argument about the paper).
by gopalv