The Economics of Open-Weight Inference

Join the discussion

• Hacker News
• The finding that flips the usual story is the sparse-MoE reversal: on gpt-oss-120b the A100 comes in cheaper per output token than the H100, because the workload is latency-tolerant and the MXFP4 experts fit in 80GB. Chip age stops predicting cost once you decouple "newest silicon" from "cheapest inference" and let price-elastic work route to whatever's efficient.

  The point about no salvage value: the paper is really pricing a rental service, not the devices, and even there the 5-year A100 term holds 80% of the one-month mark. The reason there's no clean resale market yet isn't that old GPUs stop earning. It's that the workloads keeping them earning (batch, RL rollouts, long-horizon agents) are the ones that tolerate latency, and the people running those tend to rent rather than buy secondhand.

  The part the paper understates: rented A100s aren't the biggest pool of already-paid-for, latency-tolerant capacity. That would be the Apple Silicon and flagship phones sitting idle, where the capex is fully sunk and there's no hourly meter at all. Same economics, one step further. The deciding factor is who gets to make the deployment decision, and open weights are what move that decision to the operator.

  by itsmeduncan
• Then why do people still believe that OpenAI any Anthropic have negative margins
  by simianwords
• This makes the case that open weight models are not necessarily cheaper than closed weight models

  But I'd still rather use them since it's inevitable the unsustainable margins of the closed weight providers lead to enshittification once they stop subsidizing demand

  I'd rather spend more today with a workflow whose underlying unit economics are sustainable and don't force me to inevitably look for other solutions once the closed weight providers start optimizing for profit and extracting value

  by nylonstrung
• The style and font choice here is awful. Nearly impossible to read in the daytime.
  by fred_is_fred
• The problem, IMO, with open-weight models is that you accustom to the capabilities of frontier models too quickly; and downgrading to an open-weight "frontier minus 2" or "frontier minus 3" model is often painful, since they feel way less useful than their newer closed-weights counterpart. To be honest, I don't know any companies using OW models at a large scale for their operations (agents or chat assistants).
  by jcmontx
• I think an interesting point is that hardware as of today still has no utility value after its reported lifetime has elapsed, which prevents neolabs and smaller labs from getting older HW clusters as the banks are not willing to give out loans against them. There is no agreed upon pricing for "expired" A100 clusters or similar.

  This is clearly not true, and we are starting to see compute markets, but only for rental prices/H, not for the hardware itself. I feel like there is some artificial moat being built here to stimulate sales of new hardware, because an H100 at 1/16th the price will have comparable dollar/FLOP as Vera Rubin.

  by augment_me
• Dark grey text on a black background: it's like they are trying not to let anyone read their article!
  by hungryhobbit