Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Fun reference I tested on 32 GB ram laptop with no extra GPU: llama.cpp: “what is ls”, almost immediate starts answering at one ~word/sec. Ask opencode with same model (some gwen e4b or something) to check what’s in its working directory: 20 min to response.by teekert
- Opencode system prompt contains a lot of stuff but even worse is oh-my-pi where their long prompt looks like random garbage hallucinated by a 2023 LLM:
https://m.youtube.com/watch?v=c_fQoDkULl0 (see around 8:00)
by throwa356262 - I'd be interested to see how Reasonix stacks up here - they seem to have spent a lot of effort on tuning prefix cache reuseby swiftcoder
- sounds fun, i'll try and add it.by nasutton12
- Nice writeup! I imagine these results change as harnesses are updated, so you'd need to frequently rereview.
I'd love to see a tiny, reproducible benchmark repo that anyone can drop on their own hardware and then run against all harnesses at once to compare the per turn prefix token count, time to the first token, experienced tokens/sec (and prefill), cache reuse % and a pass rate on a deterministic set of small tasks. I think it could also be useful to have some way to share results and hardware for others to compare.
by humbleferret - this is in the repo in benchmarks/matrix for most of these metrics.
uv run python benchmarks/matrix/run.py setup. i'd happily take your hardware's numbers!
by nasutton12 - Neat article.
“Chad” initially looked interesting but the minute I saw the ai-written markdown and giant commit I just left. I just can’t bring myself to read someone elses’ slop, regardless of performance.
If all a developer hand writes is a truthy and readable markdown document, I really don’t care if the rest of the project is vibe coded, but I struggle to get interested in AI generated summaries and docs.
by montyanne - Not to mention it's yet another bash does everything harness which compares itself against claude code when the closest competitor is pi.dev?by imtringued
- I for one am excited to learn more about how it spreads up to 50% between nights, and how nothing between the lean arms is a finding.by CGamesPlay
- Really surprised to not see jcode in there which beats everything else in efficiency by miles with one tiny rust binary.by lrvick
- What is with the website though? Rubbish scrolling. Junky rendering with artifacts if you scroll fast.by asdfsa32
- sorry, its a notion not a real site.by nasutton12
- Yes, scrolling with the arrow keys does nothing for several presses, then jumps, then acts normally, then jumps again. Really stupid and annoying.
Also, Alt-Left Arrow doesn't return me to the previous page (Hacker News) and I have to use my mouse.
by gramie - Notion is a "knowledge database" with awful performance and jank, that some people have decided sounded like a perfect place to host their blog for whatever reason. But these shared pages been as buggy as the first time I saw them years ago, not sure what they're doing.
- I can see this pattern of many people using Qwen 3.8 27B for local inference both on Apple Silicon and x86. This implies the model must be very good, given all these peoples' opinion converges on it.by larodi
- I've played with a fair few local models, Qwen 3.8 27B is the only one I've currently kept around (Muse Glimmer came the closest but will vacillate wildly if pushed, Qwen doesn't seem to do that (as much), I don't use it to generate code, it's essentially a glorified linter, it's pretty good at spotting dumb stuff you miss when you proofcheck your own code, especially things like comment drift where I alter a function and it makes the comment slightly (or sometimes wildly) out of sync with behaviour.
It's fine for that (and I happen to have a 24GB VRAM GPU anyway since I game on the same PC).
It's neat but for me not world changing.
It's also just fun to be able to poke stuff and see what it can and can't do (but I could see how it could also become a time trap in cases where it gets kinda close and you want to fix that).
by noir_lord - For it's size it's pretty good. I run it and the older Qwen 3.6 35B A3B, which is significantly faster, but also a bit dumber, on both my PC and my MacBook.
Wouldn't trust it for long form coding, but for shorter stuff it's really good.
by Scarjit - Qwen 3.6 27B and 3.8 27B are the darlings of local inference at the moment.
The only other thing anyone is using is Qwen 3.8 Flash Next, only by memory-rich people.
Depending on which benchmarks you believe, these models (and the Ornith 1.5 finetune of Qwen 35B-A3B) are competitive at about Opus 4.5 to 4.7 level. That matches my experience in real tasks over the last few months.
Not bad for something you can run at home for a couple of thousand dollars.
by suprjami - I've made https://maki.sh for use cases such as thisby tontinton
- Do I understand correctly that in your harness models don't call tools and pass output of one to another via context, but instead code whole pipeline as small on-demand tools and see only final output?by nopurpose
- Isn't using it with GitHub Copilot a violation of GitHub terms of usage, due to usage of internal API? I read that they have an official partnership with OpenCode, and I guess it is fine to use it with OpenAI Codex, but what about other coding agents like maki?by frafra
- HN seems to have had a stream of agent harness benchmarks floating past. And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test? Because right now nobody seems to bother testing mine! (https://juggler.studio)
I know Juggler's very new, but there's so much churn going on in this area that it's hard to know where I should be pushing it. It's hard to guess whether juggler's strengths would played well with a particular test like this, or made it look bad, all feedback about the kind of parameters people are interested in is useful to know when I'm deciding what to optimise.
by julesrms - I'll try it.by nasutton12
- > And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
It seems we need a place where harness creators can pay for that work. There are more people talking about their own harness in this post than talking about the post.
by owebmaster - > And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
by kouteiheika - A bit off topic because I'm not using local models, but I recently benchmarked codex vs pi vs omp with my workload and found codex to be both faster and more token efficient than pi/omp. There was not a single case for which pi was faster/cheaperby toasty228
- The trick to use Pi is you use it in N ways if you have N use-cases. You need N set of shortcuts/plugins/aliases etc for that. Can get tiring at times.
So I keep Pi for just one case - when I have to easily strip everything out for some work. Anything heavier and OpenCode or Claude are ones. I am sure I can make Pi behave as I've suggested above the "N harnesses within 1 harness" and I even tried but it simply started getting out of hand and using the harness started becoming the frustrating hobby.
As for OMP, I just don't understand why would anyone use that not Pi or other "full-fledged" harnesses.
- Pi is a very basic harness by design. On the other hand OMP is a bloated mess of other people’s workflows.
The trick with pi is to extend it yourself as you use it. It’s pretty easy to do.
by weiran - What is this supposed to mean?
"it spreads up to 50% between nights, so nothing between the lean arms is a finding."
by alex_john_m - Means Claude can't write for shit.by stavros