Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- man i can't say how much i used to like cuda when i had a nvidia gpu it made ml so much fun and on amd its a war especially on rdna 2 cards which i have. hopefully one day we will be able to properly translate cuda for its amd counter partsby Nurysso
- I can only hope NVIDIA won't send an army of lawyers against this.by maxlin
- If anyone has used this, how good is this? And will it allow running AMD GPU with MATLAB? I have a 7900xtby harhargange
- I made cuda-metal btw (for mac kek), https://github.com/lulzx/cuda-metalby lulzx
- What models can it run?by sroussey
- I wish there were a way to use RDNA1 cards with CUDA for AMD. My 5700XTs are sitting in a drawer.by system2
- RDNA1 isn't good for a whole lot, even flagship RDNA2 cards are a stretch for many things. The lack of WMMA/matrix multiply/BF16 is too severe of a penalty.
The FP16 throughput on RDNA1 is both shader reliant and requires everything to be packed first. Even with 2 or 4 or 1000 cards, you would be consuming all of the available memory and memory bandwidth just packing and unpacking values, and if you really want to dump a hundred billion tokens into making it work anyways, you're only going to find out that even if you bother to sit there ferrying packed values to ram or disk before then issuing the instructions, paying that already severe penalty again when the values then have to be unpacked is so steep of a cost that the 256 BF16 flops/cu/clock's effective throughput is outright lower than simply doing it on a Zen 2 processor. You also don't have INT8 (or really INT4) on RDNA1 so the other RNS/CRT tricks aren't viable.
Sadly RDNA1's VCN2 also lacks actually good x264 bframe encoding support, or even P010 for 10 bit color, so what I'm saying is you should sell them. Used Radeon VII's are like $260, you'll go a lot further with those especially if you throw in a 7900XTX, and then augment that further with a 9070 CRE (you only want it for its int8 cores), and of course 128GB of ram.
E: And sure, that's 3, or ideally 4 GPUs, and a good bit of extra work. But that gets you up to more than halfway to the naive performance of a $15,000 MI300x in a surprising amount of cases, with additional strengths that it lacks. For far less than half of the cost
- There are also interesting efforts like:by latchkey
- Interesting option for CDNA architecture chips. I wonder if this moves to an open standard?
AMD GPUs build for AI specs for reference: https://flopper.io/gpus?vendor=AMD&page=1
by triwats - To save everyone a click: No cuDNN and based off an ancient version of ROCm for windows (7.1 has been out for ages, 7.2 is current.)by KennyBlanken
- No, 10.1 is currentby woctordho
- > CUDA for AMD on Windows
This gives me strong "Bash on Ubuntu on Windows" kind of vibes
by qalmakka - Right, it reminds me of this website back in the day, when people really were coding these things as well instead of just generating 100% of it. Being sarcastic here, but stuff like "How I coded Java with Rust" were a common thing hah!by Shocka1
- I think projects like this one are a clear indication that CUDA *is* the de facto standard and that aspiring open standards are competing for their niche. The rest of the world is still living in the CUDA ecosystem. Asking accelerated HPC devs to port every project to a different language (and possibly maintain two codebases) is something we haven't asked of anyone else for decades now. And yes, LLMs can help (although CUDA makes up the overwhelming majority of their training data, with the hilarious result that you need to give them a disambiguation table like "you'll want to write $CUDA, but write $else"), but AI is clearly not the correct tool for the job. A compiler is.
ZLUDA is JIT (it's working with the binary, not the source, which usually means you've lost some information) and it doesn't support CDNA, which is a showstopper for enterprises, tho it's a great project and it's awesome seeing it work on Windows (I wonder whether MS Defender complains about it due to library injection? IIRC, it used to. That'd be another enterprise showstopper)
Someone here mentioned scale-lang.com already as a portability tool - it also beats HIP in the performance game, on average. scale-lang.com/s/discord is where its community lives.
disclaimer: I am part of SCALE's team, Spectral Compute
by atsetilam - AI will take down Nvidia’s moat. When it becomes trivial to translate CUDA/PTX to HIP, SYCL or Metal, CUDA is no longer the moat, it becomes the intermediate representation.by swerner
- yeah yeah, "when" an often keyword with AI it seems. As Mr. E. Nigma put it - what always comes but never arrives? Meanwhile the moat deepens and it's build on inertia and laziness and Nvidia knows this really REALLY well.by Keyframe
- > When it becomes trivial to translate CUDA/PTX to HIP,...
ZLUDA is already doing that, no?
by bayindirh - i swear people who are outsiders here have only clickbait takes; if you've never had to ship GPU code professionally you should just not comment on these things.
the source language has never been the moat. Nvidia sells to hyperscalers. Hyperscalers have armies of kernel authors who have no issue translating shaders by hand (or now with claude). Nvidia's moat is (and will remain for the foreseeable future) the entire stack. you cannot fathom the pain and misery of working on literally any other stack. if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf you really have no idea what you're talking about.
EDIT: i can't believe this really requires saying but graphics and compute are not the same domain at all. if you work in graphics for GPU but not compute then you are still way out of your depth commenting. to wit: graphics people do not (and cannot) write CUDA kernels/shaders.
by mathisfun123 - trivial to translate (or transpile) - okay. trivial to understand the result - not so much. trivial to then evolve it - hm... perhaps a different story. still, it seems very likely now, that such "quick rewrites" are viable, not sure if an open approach to them is viable. a newly born open project that was LLM-derived, and not by a credible author, which spans hundreds of files no human eye has ever looked at, can only work for a closed organization, but will never be trusted by the general audience... just like that.by larodi
- Off-topic and somewhat of a rant, but I'd far prefer us all focusing on open standards like HIP, SYCL, OpenCL, etc.
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
by linuxhansl - yeh - exactly.
Sucks like important libraries like Alphafold are locked into CUDA. Its ridiculous for researchers.
- it will only get worse now when hughingface was bough by nvidia, not immediately, but in span of year or years...by kiicia
- Commercially it's better to have AMD support CUDA which can help break Nvidia soft monopoly.by HeavyStorm
- It's impossible to have an open standard. Hardware accelerators are nothing alike and have different perf characteristics. Each kernel is tuned to the hardware. The idea of writing a performant kernel in opencl is a fantasy.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
by anon291 - I would too, but sadly that's Khronos' job to organize, and they've had trouble getting American vendors to work together.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
by bigyabai - The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.by mschuetz
- On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.by mistercow