Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Great
    by PrimeAli
  • Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params?

    BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

    by sylware
  • Yes, ROCm can be used to run frontier models and is being used by OpenAI, Anthropic, and Meta.
    by wmf
  • Kimi-K3: 2.8T

    Qwen3.8-Max: 2.4T

    DeepSeek V4 Pro: 1.6T

    DeepSeek V4 Flash: 284B

    (all are total parameter counts, not active parameters)

    by wren6991
  • This is still quite a bit away from the performance that deepseek gets on their H800. In their DSpark paper they report a throughput of 15k tokens/s/gpu. The MI300 should be able to compete with the H800 so there are probably still quite a few optimizations that can be made.
    by xorfish
  • throughput scales superlinearly with number of GPUs when networked well and deployed with wideEP, so 1x won't compare.

    also it would be interesting to figure from the DSpark paper whether their numbers are consistent with the GPUs still being H800s, since they never actually say...

    by somnial
  • Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”.

    Dumbed down quantization?

    No. Full intended inference weights preserved, so far so good.

    Slow performance?

    No again. Looks like you could get over 150 tokens/second.

    Give up context window size?

    Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this range, and quality does start to drop off toward the full size.

    by WhitneyLand
  • To be fair, in most of the headlines, x is a Xeon from 2017 or a macbook pro or a gpu with 4 gb of vram. In this case, it's a datacenter gpu, and one that actually has more VRAM than the NVIDIA equivalent.
    by throwawayffffas
  • I am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.
    by bwfan123
  • In my experience the 1M context is genuinely too much. The first time I swapped from OAI to DSv4P, I checked and double checked that the harness/etc was working correctly over the course of hours and hours of work thinking that I had set something up wrong because it simply never had to compact! The drop in quality is arguably less than that of what you get from compact to compact on Codex, which is good for what it is or was.

    Was also surprised to learn just how much of Codex's window was being burnt on shit I didn't want or use. Sure I can pass this and that flag to eliminate most of it, but for a $200/mo product aimed at professionals, that isn't something anyone should have to janitor (also totally ignoring the bandaid of banked resets they've slapped over their repeated mistakes).

    It's wild just how far $20 will get you with Deepseek, even at their new rates. Buyers Remorse is my very least favorite feeling, I felt sick thinking about what the $1200 I had given OAI this year would have gotten me had I only tried sooner.

    by monster_truck
  • nice! i think the higher HBM on Mi300x is really useful for this kind of thing

    we did some work on this for 2xMi300x (kindly referenced in the readme) https://blog.doubleword.ai/deepseek-v4-flash-mi300x. https://hotaisle.xyz/quick-start hotaisle is great for getting Mi300x to experiment with

    by fergusfinn
  • Strange that in the prior art they didn't list DwarfStar, as it is able to run the same model (probably quantized differently though) in less memory. Maybe the author isn't aware of it?
    by GTP
  • The focus of this repo is the MI300X and DwarfStar doesn't include any optimizations/fixes for it.
    by MaKey
  • Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB.

    Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

    by Tepix
  • Addendum: I was wrong, you‘ll need two of these cards.
    by Tepix
  • Just want to add that while the MI350P is a PCIe card, it is designed for servers. It just has a heatsink (with no fan) which the powerful full-case fans of a rackmount server are supposed to cool. So while the MI350P is certainly more attainable for us regular folk due to its formfactor, we won't be able to just drop it into our gaming PCs like a regular graphics card.

    That being said, if you're dropping tens of thousands of dollars on graphics cards then picking up a rackmount case to go around the card is pretty insignificant.

    by craftkiller
  • How do you figure that?

    When they just loaded the weights alone, it was taking 156GB in vLLM. After warm-up and adding a KV cache pool, it took over 200GB.

    And this implementation is already cutting down the 1M token context window you would normally get.

    by WhitneyLand
  • I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.
    by majke
  • Give it an AI-bubble pop and these will be flooding the market.
    by baalimago
  • I thought it was a consumer grade GPU until I saw the 192GB of HBM and 256GB or RAM.
    by _joel
  • You can get one on ebay for like 20k, but it comes without the backplane and i dont think there is a pcie card adaptor from china like the ones for h200.
    by throwawayffffas
  • The MI350p exists and should run a decent quant (say, the ~96GB antirez mix) well, but you can get two rtx pro 6000s for one of these, or 8x (actually more) r9700 + probably the gear to run them, etc.

    Otherwise, you can probably buy one of these second hand from somewhere (SXM A100s are available that way) and run it in an adapter board.

    by Lwerewolf
  • It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud (https://www.amd.com/en/developer/resources/cloud-access/amd-...) powered by Digital Ocean at $1.99/hour.

    Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to generate $3-$4 worth of tokens per hour and still maintain a usable per-user throughput.

    by zhoutong