Performance per dollar is getting faster and cheaper

Join the discussion

• Hacker News
• But RAM prices skyrocketed!

  The AI companies owe use money. As does e. g. NVIDIA for becoming a cartel.

  by shevy-java
• (in a high-pitched, pathetic regency-era British orphan voice) Please sir, may I have some compute as well?
  by bitwize
• Can you actually rent an MI355X per hour anywhere right now?
  by ilaksh
• world is not limited by Nvidia, AMD can be used
  by gowthamsaiyadav
• then why is token per dollar getting more expensive?
  by calin2k
• Because lots of people are willing to pay more dollar for smarter token.
  by FeepingCreature
• There are a limited number of these available in comparison to demand. I think people figured out that LLMs and VLMs can do real work that can replace a lot of humans. And for plenty of jobs, it's good enough to reduce already outsourced staff by 75-90% at a fraction of the cost.
  by ilaksh
• Because they are dumping/subsidizing it token processing to try and get companies to fire as many people as possible. So they'll be dependent upon the companies when they have to Jack the rates
  by AtlasBarfed
• What is a knee, in performance talk?
  by hahahaa
• I used to be high-performance like you, then I took an arrow to the knee?
  by nnevatie
• A place where the slope/derivative/incremental-performance-per-price changes.
  by kgwgk
• No word on what this actually means as a consumer. What's the price. Is it lower than NVIDIA serving?
  by killingtime74
• They seem to be serving it at 3x the price while also struggling with maintaining uptime on openrouter; while the vercel router advertizes even bigger speeds but has no clear uptime stats

  I guess you really do have to try it at least for some time to actually know

  by mixtureoftakes
• *especially as many currencies weaken
  by conorcleary
• This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production setting. It's only interesting to hobbyists that want to run the model locally.

  It's genuinely neat that AI can find the right optimization pathways in an AMD inference server to unlock this but at the same token (pun-intended) this is a classic case of benchmark hacking that doesn't stand up to real-world application.

  by beffjezos
• hi yes it’s not optimized for single stream it’s optimized for total node throughput
  by technoabsurdist
• You got it backwards; it's ~200 on single stream so the 2,600 is achieved with ~13 streams.
  by wmf
• They fail to mention non speculative numbers & whether baseline was nvfp4 as well. So much for erosion against an older gen
  by villgax
• I like the metric of tok/joule a lot. it really brings to mind a lot of really nice ideas about energy and work and ideas and thought and efficiency
  by sometimelurker
• I'm interested if anyone knows how much legwork the assumed 60% cache hit, plus running a quantised model is doing? Esp. compared to what the headline half implies is a full fat GLM5.2
  by alienbaby
• So... the headline is about performance per dollar per dollar?
  by BurningFrog
• yeah but we are still far far away from being able to run the frontier model equivalents locally without significant quantization

  even having something like opus 4.8 locally would completely change the landscape

  by zuzululu
• Isn't this pretty much a given? Performance per dollar has to be a ratcheting function because how would something more expensive replace something less expensive?
  by gcanyon
• That sounds literally impossible.
  by johanvts
• Agreed. The writer is pretty loose with their comparisons:

  * What does it mean for "performance per dollar" to get faster? Higher, maybe; rise faster than it has in the past, maybe, but just "faster"? Nope.

  * The article cites some equipment as being "2x cheaper". I think they mean "half the cost", but if so they should say it.

  by dtgriscom
• The compute-in-memory and neuromorphic paradigms are likely to push this much, much farther over the next decade as more radical improvements make it out of the lab. Sooner or later it will involve new materials and new nano devices and providing multiple orders of magnitude better efficiency. And just scaling up existing things like MRAM.
  by ilaksh
• Slight criticism of the headline there, you can't get cheaper per dollar.
  by adammarples
• Do these providers have 80+% gross margins or is something eating into them? Maybe utilization?
  by oDot
• hi i work at wafer. no the margins are lower averaging at about ~40%. utilization is one of the highest order bits in determining margins here, yes.
  by technoabsurdist
• I was hoping they would be discussing some path to improving things faster and cheaper. But in this post it looks like they offer quantized version for the same price as full version, and a fast version at much higher cost.
  by mchusma