Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- How is Pi so efficient? You'd think agent harness made by model makers would perform better.by nkzd
- they spent very little time validating what they're doing, and it works by not doing much of anything. If you spent a month figuring out a specific model+harness, you'd be way more efficient.
Other hanresses are doing overkill so they can work with any model.
by cyanydeez - I think in general the model makers and to some extent their clients want a slightly higher pass rate over efficiency. This makes sense: for critical first week impressions clients notice pass rate much more, and only later start to grapple with cost. For example this is why High is the default reasoning for Fable, not Medium, and that choice of priorities propagates throughout the stack.
- Claude code's system prompt is filled with irrelevant stuff about how CC works, so that the agent can help the user set it up. And there's no way to disable all the extra stuff, AFAIK.
There's https://github.com/skrabe/lobotomized-claude-code , which strips many of those, but I'm not sure if it is "legal" to use.
by el_isma - Different incentives. Claude Code makes more money for Anthropic by generating larger contexts. Anthropic also recently changed their tokenizer so the exact same code input creates 30% more tokens, so there's a pattern there.
- Seems like for a hobby project $1 or $2 per task would add up a bit, depending on how many tasks you need to do. I mean it makes sense for a software companyby pianopatrick
- It seems that pass rate decreases with effort increase, on GPT5.5? This is highly counter-intuitive and I don't see any explanation, any idea why they'd get this result?by virgilp
- Looks to be within the realm of natural variance expected from naturally variable models, ie error bars.
- Would be great to see time spent per task per model. Especially since article references 390+ tokens per second for GLM5.2.by yigitcan07
- 1) Many models are now competitive at the top tier, including open source. 2) GLM 5.2 in particular was a major step forward in open source coding agent performance, 3) Harnesses make a huge difference in cost-performance. 4) Cheaper per-token does not imply cheaper per-task.by falaki
- Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.by falaki
- Anthropic is not beating the charges that they inflate token consumption with their own harness given these findings that Pi is 2.2x more efficient at token management. Big "toothpaste ads tell you to use way too much toothpaste" energy.by Schlagbohrer
- Could it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher?by lukax
- they explain this is a benchmark, all models/harnesses receive the same promptby rootatixww3
- Wow!
It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results.
Definitely a few surprises here:
1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25 per task vs $2)
2) Absolute best pass rate (90%) was from Opus 4.8 x-high using Pi, beating out Opus 4.8 using Claude Code
3) Pareto frontier performance from any of the models (Opus 4.8, GPT 5.5, GLM 2.5) was using Pi rather than native harnesses
Apparently Pi used 3x less context than Claude Code, and one takeaway is to use Pi regardless of what model you are using. The other takeaway is that in real-world performance GLM 5.2 is the equal of Opus 4.8 unless you run Opus 4.8 on x-high in which case you can eke out a 2.5% increase in pass rate at the expense of doubling your cost over GLM 5.2
- The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.by zaphar
- > the results showed clear clustering of the models and harnesses into 3 capability tiers
pretty sure the only thing making that 'clear' is the coloured stripes, if you took that away it'd look like two tiers
good result for GLM 5.2 though
and Sonnet 5 seems like a waste of time
by anentropic - This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.
I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.
Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.
Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.
by cpard - I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?by yodon
- for that you would need to compare the same task implemented in two different languages - C# and Python for example, no?by rootatixww3
- I don't have hard data, but we have shifted to Rust and Swift (for frontend UI) for the bulk of our dev simply because it is a lot more predictable, easier for tool calls to edit, the build steps produce easier output for the agent to loop on, the tests are easier to write/get results from, etc., although I am mostly measuring this in time, not cost.
Once the thing is rock-solid it's relatively easy to do a Swift->HTTP/HTML/CSS/React/TypeScript conversion.
by trollbridge - We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting virtuous loop. I didn't expect this outcome: for whatever reason I assumed proprietary harnesses fine tuned to work with a companies' models would work better? ps/random aside: there is something slightly off about Pi's edit command, we are planning to investigate this further and patch this as we have quite a few session traces now..by redmalang
- Can you explain context aggregation here more? What is that? What are you doing and how is it helping?by Schlagbohrer
- > we've been able to automatically aggregate context from pi sessions
what do you mean by this ? do you rewrite the context in your proxy ?
by Sphax - I keep hesitating to use Pi because I primarily use Claude and I worry it will be much more expensive to pay for API usage rather than the bundled subscription usage. But I haven't actually evaluated the cost tradeoff. Should I get over this hesitance and dive into Pi?by sanderjd
- Aside 2: Anecdotally we found that Pi performs more or less on par with native harnesses at lower cost on decently specified prompts. It is also phenomenal at context cacheing especially on Deepseek models (its hard to precisely attribute credit here are my understanding is this is a DS speciality). But it fails much worse on poorly drafted prompts. I'm generalising but native harnesses seem to be better kind of flailing along on those.by redmalang
- Yes, this is a known issue. A significant amount of Edit tool calls fails in Pi witg newer models.
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
by lukax