Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I am convinced that the combination of capable open weight models and specialized hardware will mean that Apple (and other hardware providers) will start shipping computers with built-in, hardwired, "LLM-on-a-chip" cards that are capable enough to meet 90% of your AI needs.
I really believe that in the near-term future we will run our LLMs in hardware, not in software. Hardwire a capable model into a device the size of a graphics card, embed it into a laptop, and you have something that uses less power, does faster inference, doesn't require additional CPU or memory, doesn't cost a monthly fee, and will probably eventually be available for under a (few) hundred bucks.
by akie - The article's own example makes the point I'm about to make:
> To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon.
That's all because it ran through the most expensive frontier model for a mechanical task that a cheaper model could easily handle.
What hardly gets mentioned is that most people don't actually measure what each task costs them on a granular level. A lot of the waste comes from running everything through one expensive model without considering breaking the big task into smaller tasks farmed out to cheaper models.
Whether the labs' economics hold is above my paygrade. It costs what it costs. What I can control is my own usage.
Like everyone else leaning in heavily on AI usage, my tokens started running out mid week...sometimes within a couple of days. I had to do something about it or double my spend. So I started tracking my cost per task type a few months ago and it completely changed my workflow. The lowest hanging fruit was the mechanical stuff. Moving that to the cheapest models was a game changer, and much faster to boot. Mid tier models take the workhorse tasks. The frontier heavy hitters are now only used for judgment calls like reviewing and planning. Spend dropped dramatically.
Freeing up all those tokens made me even more ambitious to explore parallel ways of working, to get even more out of what I was already paying for.
by jarodrh - Would prefer not to offend the author, but I do believe this article has very little for the HN audience. No new insight, and no numbers or new information.by jeswin
- There is a wave of users switching over to DeepSeek Flash. There are Reddit threads of users sharing billion token spend for $20.
If all of global spend on Anthropic/OpenAI/Gemini APIs just switches over to DeepSeek then easily we can decrease total AI spend by 10x
by arjunchint - I have already seen a number of people doing the math on what it would take for hardware to self host a Q8XL quantization of GLM5.2 shared between N numbers of people.
There's additional advantages that everything you query, all of your context cache and everything it outputs stays private and can't be arbitrarily turned off by external interference.
Personally I think it would be a fairly good bet that something with the 1TB of RAM needed to properly self-host GLM5.2 will still be a very usable piece of hardware in 4 to 5 years from now. There will be even larger, newer models available, sure. But there will also be better models that continue to fit in the same size.
by walrus01 - I am using perhaps 15% of usage count on Claude with just the normal subscription. And I do full time software engineering and would say I use quite a lot of AI input on thoughts, designs and code drafts.
So how these companies and people manage to use these absurd amount of tokens is a mystery to me. It feels like this are just running huge amount of non-vetted data to the LLM's and or running loops against the LLM's which only produce fractional results if not wasted results for insane cost.
So really it is the equivalent of just burning money, or heating your house in the winter while having all your windows open.
by yturijea - > To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon.
If you can use a subscription with any of the SOTA models, do that.
Instead of around 4k EUR in token costs, my Opus usage costs me 108 EUR (with taxes) per month with their Max 5x plan. It's the same with OpenAI, those are heavily subsidized.
It doesn't make sense to pay per-token, unless you must.
> What is happening here is that leading AI labs are charging not only for inference but also for research in model architecture, training data collection and curation, model training cost (which can be tens or even hundreds of millions of dollars), paying their employees and recovering the marketing costs.
Chances are, they're never getting that money back. Best case scenario, the hype around AI slowly declines, worst case - it crashes and takes a part of the economy with it.
Also anyone doing distillation with hundreds or thousands of those subsidized attacks is probably winning big. Especially as the model architectures (e.g. DeepSeek V4) are more oriented towards efficiency.
> Last but not least and in fact the most important factor, is the ability of users to run local models. So far, almost everyone is using cloud-hosted models and local models are either too big to deploy or too slow to work with. With advancements in chips, this will change in 4-5 years’ time.
Currently beefy hardware to run them fast enough to be competitive with the cloud (at least 60 tps) is expensive and even then the small local models quite suck compared to SOTA or even DeepSeek V4 Pro and GLM 5.2, though they're way better than they used to be (compare Qwen 3.6 with 2.5 for example).
by KronisLV - The problem space has a few aspects:
1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs.
2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen.
3. We're massively overusing SOTA models. As long as you're on a subsidized subscription, you can use Claude Opus 4.8 high to write blog article meta descriptions. If you paid by token, you wouldn't do that.
4. Open models are a wildcard that could completely change the calculus.
by FinnLobsien