Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > Using harnesses that are “less chatty” (more token efficient), or tuning existing harnesses to generate less token overhead.
agree, this works, undervalued!
look at minimal agents that protect the context window:
some thoughts on the other tips (for coding):- pi (https://github.com/earendil-works/pi) - smol (https://github.com/smol-env/smol)1) stronger models are more token efficient for open ended tasks because at the limit …
that of course is difficult to benchmark, so most attention goes to simple benchmarks that show cheaper models can get similar results on 'closed' tasks with easy to 'eval' results- stronger models can solve tasks that the weaker models can not solve - stronger models make fewer mistakes, compose things better (cli, abstractions, …) - navigate the code base better - are better at removing and simplifying the code base again2) dynamic request and task routing sounds great/obvious but is very very hard
using cheaper models and auto-routing do work well for 'closed' tasks where you have something repeatable and can evaluate whether a certain quality threshold is reached that you are comfortable with- to benefit from caching you don't want to switch model or inference endpoint - to _know_ a certain request can be routed to a weaker/cheaper model needs good context and a strong model to get right and often is still unknowable because the active coding session can go many ways and turn from trivial to challenging in a few turns, always in motion is the future, if you get it wrong you are back in the problem space of #1for open ended coding sessions it is not so easy
that said: cheaper does not have to mean weaker, you want to look at the pareto frontier and stay up to date on new good models
there are many models like deepseek v4 flash and luna that are both cheaper and way better than most other models
by tosh - > nearly every company deploying AI tools at scale has hit the same wall: exponentially growing costs. That curve is unsustainable - left unchecked it will eventually overtake revenue.
But the question remains, AI hasn't shown any explosion in growth/revenue for most companies. The effects seem muted.
- If only one company had AI and nobody else did, then that one company would be expected to start seeing an explosion in growth/revenue. As it is, everybody has access to AI, so extensive use of AI should just result in keeping up with the competition. The ROI to spending on AI is "not dying".by williamstein
- “agentic coding has measurably improved every velocity metric we track and, in some teams, driven an order-of-magnitude gains in output”
Of course it has. But is that a good enough measure of value? What about the business and customer outcomes from all the extra lines of code and closed JIRA tickets? Was it worth it?
by desbo - This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs.
Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
by bisonbear - The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks.
In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
by pwendell - Surprisingly pragmatic and info packed article..
Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools.
I think building companies is going to look more generic in the future because intelligence is an API now.
by nichochar - Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be good for someone to just systematically write down what those are so that others can try out the techniques if they find them useful.by pwendell
- > "Exponentially growing costs"
I don't think people thing don't think enough about the exponentially growing costs with AI. The cost of tokens, the cost of bugs, the cost of technical debt, the cost of software bloat and slowdown, the cost on the RAM market, on the environment, the costs of the little lies and hallucinations that creep into our codebases, blogs, books, and search results.
What seems like a manageable margin suddenly starts growing and destroying everything.
by adamddev1
probably where the 1 trillion+ dollar valuations are coming from.... lots of thick saas margins they can siphon/skim from the whole industry.> What seems like a manageable margin suddenly starts growing and destroying everythingby andrekandre- I find this funny and interesting at some levels
1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes
2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model
Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.
by sandeepkd - buddy, they're on enterprise plans paying per tokenby oh_no
- Open ai is allowing subscription use, anthropic also paused the effort to stop subscription use.by pinkgolem
- Databricks will be using the API anyway, thats all you get with an enterprise agreement.
- ... where in the article did they say they were using subscriptions? I'm fairly certain enterprises can't access subscription pricing in any case, they're all API costs (Anthropic doesn't support more than 150 on subscription pricing [0][1]).
[0]: https://support.claude.com/en/articles/9797531-what-is-the-e...
[1]: https://support.claude.com/en/articles/9266767-what-is-the-t...
- What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant.
One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)
by dgellow - Do you think Anthropic or OpenAI will eventually try to crack down on routing harnasses? Provide a more vertically integrated experience? They are already trying ro ship hardware products.
- Careful. If you admit to using models that weren't trained by OpenAI or Anthropic then you might hauled in front of Congress: https://www.scmp.com/news/china/diplomacy/article/3362616/us...by platinumrad
- That is a wild article! Hopefully the recent US open source model initiatives can pan out to cut costs, because I’m not a fan of dependence on Chinese models either, but I’m in a spot where I need to maximize intelligence on minimal cost…
There’s not even a distant comparison US lab rn which is sad
- Why would it matter if foreign companies analyzed DoorDash data? Pizza deliveries to the Pentagon is all I can come up with, but that's publicly available at https://www.pizzint.watch/
I would bet my entire Polymarket balance ($0) that some military contractors have already asked AIs on the public Internet to design software for them.
by axus - I'd prefer congress to be asking questions (this is all they are doing so far, based on the article) before doing any legislating.
- I suspect that when it comes to hard complex software products, you’re better off ignoring agents and doing “trad coding”. What you lose in short term speed you gain in manageable complex codebases.
If you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term.
Now of course, there are products that just involve lots of code but are not actually complex. This is generally the project with like hundreds or thousands of features but most of the features are separate and don’t actually interact in complex ways. Think a task management app with hundreds of features like calendar, email integration etc. there I think agents gives you more bang for the buck. Just my thought, using agents at work.
by sashank_1509 - > if you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term
A bold claim to make with little to no supporting evidence
by SOLAR_FIELDS - I work on and maintain a 900k+ LoC codebase with mixed languages, runtimes, deployment models, cloud and airgapped ops and it's 99% written by AI. The difference I see the most between our approach and others is that we see the end-user product as an output of our context engineering internal machine, we focus on building that machine and we measure the output like madmen across all and any qualitative software metrics and signals (traces,collectors,otel,snapshots,profilers,custom debuggers,videos and more). For any change we do, less than 5% of wall time is an AI doing reasoning or coding, 95% is running the verification deterministically. Of that 5%, about 90% is spent on non-feature aligned code.
Velocity and quality is through the roof, "10x" would be a regression at this point, the bottleneck is only figuring out what is relevant/good to deliver, the implementation is the output of the machine.
by lazarie - There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year".
On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.
by lbriner - > On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering.
Engineering payroll needed to be slashed and the C-Suite was happy to let their managers get some lashes in on the uppity workers.
The point of AI is:
1. It's bad engineering and everyone knows it.
2. Management is pushing it through anyway.
3. Deal with it, asshole.
The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control.
Edit: It is just as much about affirming the exalted status of management in comparison to labor. It's a bitter reinscription of the status hierarchy in tech; Management is good and labor is bad; Management is considerate and labor is ignorant; Management is strong and labor is weak; Management is superior and labor is inferior.
by dan_q - On this planet?
They’re not saying they regret doing it, or that it was a mistake.
They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized.
Pretty standard business or life iteration.
by therealdrag0 - Weirdly a lot of the come from company that sell Ai credits in some capacity, and who are also selling (or will soon) some kind of AI gateway or routerby K3UL
- Someone did notice, as they panicked at the cost.
What didn't happen is any analysis of cost/benefit up front. Many of the corporate decisions around AI have seemed characterized by companies blindly copying each other.
by nitwit005 - The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as expensive, there are some changes to the harness (we found randomly Claude Code and other harnesses will make changes that drastically impact efficiency), and then maybe you have some organic user growth as well and BOOM suddenly you're at a $10M run rate within 60 days. And it's now impossible to forecast future growth.
It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations.
The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.
by pwendell