Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It’s not meaningless at all: every query returns usage and I can calculate the cost.
EDIT: this is like saying hourly rate or salary is meaningless. Different people have different output. You have to evaluate performance.
EDIT2: just pray the LLM providers don’t start taking Patrick McKenzie’s advice and start charging based on “value delivered”
by dools - Can you really calculate the cost easily ? Given most of it should be reduced by input caching read (if you don’t want to have a crazy bill)by ttoinou
- As well as cost-per-task I think it's worth thinking about speed, especially in non-coding contexts that benchmark less cleanly
We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)
(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)
by dbuxton - Concrete example: I’ve been trying to use Claude to generate all my commit messages, but it takes 5-10x longer than if I just write them myself. Mine are less detailed, but one line changes are sometimes inconsequential (especially white space reformatting). I wish there was a model that understood the codebase well enough to generate commit messages in half the time.by sillysaurusx
- Not every application uses LLMs the same way. For some use-cases, price per 1M tokens is absolutely meaningful. Eg, we do a lot of pretty basic classification/entity-extraction/summarization type work on large inputs (100k tokens per request being very common). It's pretty easy stuff; Gemini 2.0 Flash was perfectly adequate, quite fast, and cost $0.10 per 1M tokens (and even less when we could make use of the batch API). Every newer more powerful model obviously can handle the same work but costs significantly more. When we're deciding what model to use, price per 1M tokens is definitely a meaningful metric.by thraxil
- Sonnet 5 makes more sense when you pretend the higher thinking efforts don't exist. (His test was on xhigh)
Anthropic's own release announcement mentioned that it's less cost competitive per task than Opus at higher thinking levels. It's significantly cheaper at lower levels though.
I'm wondering if this is going to be a universal pattern of smaller models: they're less smart, so to achieve the same benchmark results they have to think a lot more and hence become expensive.
Benchmarks force models to solve the problem entirely by themselves, requiring thinking. But if you pair them with a smart model (who thinks and solves beforehand) they won't need to solve the hard parts and can run on low/med. I suspect that was Anthropic's intention.
by usef- - Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
by janalsncm - Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
- Tokens in a casino is pretty accurate if you think about it. You never really know what you'll get so it's tempting to "roll" over and over, thinking every roll puts you closer to a bellringer. It can even get addicting for some people.by pooploop64
- The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.by nathanyz
- Sonnet 5 is a huge regression and many times it performs worst than deepseek. I believe Antrophic staff themself don't use Sonnet and use Fable for everything.by cute_boi
- Efficiency is the next frontier in LLMs, and I'm not confident the American companies are taking it seriously enough. DeepSeek, even in a naive API-calling loop, serves something like 80-90% cached tokens at an absurdly low price per token. Using an agent harness tuned specifically for their caching (Reasonix) pushes the cached tokens to 97-99%. DeepSeek is consistently among the cheapest models per task in my benchmarks, while also performing quite well. I'm still almost always using Claude for work, but for side projects, small stuff, etc. and anything better served by an API rather than starting up Claude Code (or `claude -p`) I'm using DeepSeek pretty often.
Anthropic models also shut down on a lot of security-related work, which is what I've been spending a lot of time on lately. I expected Fable to refuse this kind of task, but even Opus 4.8 refuses to build a verification harness for security bugs, as that involves exercising a discovered bug to prove it's been fixed in an automated red/green way, which looks like exploit creation to Opus' guardrails. So, I have to use other models for that work, now, though most of the original benchmarks I built were built with Claude.
by SwellJoe - In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
by kpw94 - I feel like we need to see more proliferation of local LLMs to start seeing ones turned to be terse, rather than maxing the amount of tokens user pays for
- Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.by c7b
- I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation.
The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.
Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.
If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.
Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.
by kfsone - One more ~~lane~~ layer of LLMs is sure to solve all our problemsby c-hendricks
- > it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on
This downplays the incredible things that can be done with it.
There's a lot of noise, yes. How long has the web existed? And yet we're still figuring out how to optimize (HTTP/3).
Disregard the signal at your own expense.
by paulddraper - Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.by yreg
- Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.by janalsncm
- Isn't the benchmark working exactly how it should in that case?by efromvt
- That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.by teraflop
- Cost per tokens is as valid as price per unit volume of fuel.
Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.
Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.
This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
by Lerc - Except, a gallon is a gallon no matter which gas station I'm at. Also I know my car's gas mileage, and it doesn't change when I visit a Shell station instead of a Chevron. The composition of the gas is regulated, as are the pumps that dispense it. There are inspectors from the state whose job it is to ensure that when I buy a gallon, I really get a gallon.
Tokenizers aren't standardized to anywhere near that level. A "token" from one isn't the same as a token from another.
by bjt - That’s not a good analogy because a gallon of gasoline has a known amount of energy in it. The efficiency of each vehicle is also known, at least in a way that is easy to compare on a relative basis.
I can go to 10 different gas stations and buy the same amount of energy from them. When I put it in my car I’m going to get the same result out. The differences are very small.
by Aurornis - I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.
For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.
But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
by Jcampuzano2 - What company is it, so I can compete with them? ;)by Schiendelman
- All of this should be abstracted by the harness, which (ideally) should figure out which model to use for which queries.by Ferret7446
- That's why you have to reframe in terms of total cost per task and factor in model token generation quantity and multiply that by the base cost of the model. Then factor in your time value if you dare. Then you should get a more meaningful business metric.by greenavocado
- > may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Well that's the problem with these black boxes. You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume "the cheaper and older model is probably going to cost less overall to use than the newer, more expensive one."
by xienze