Compute Ecosystem of AMD GPUs

The following post is an anon rant about AMD's compute ecosystem:

I work at a cross-platform scientific software company that has significant investment in developing products for GPU-accelerated computing. I joined this company just before the company adopted CUDA as a backend for compute, more than 10 years ago.

At the time, NVIDIA was new to the game, but because of the willingness of NVIDIA to respond to our feature requests, plus their technical help to develop code quickly, we were able to develop some semblance of a production-ready minimalist CUDA backend within a year or so of starting.

Over the years, we were vary of being over-reliant on one GPU vendor, for both business and technical reasons and so left spaces in our code for if we ever wanted to expand outside of CUDA. Every year, we would assign an engineer to look into alternates. For many years, the most promising answer was OpenCL. We would try integrating OpenCL into our code base to run on both NVIDIA and AMD's hardware, but either the libraries were lacking features we needed (features that NVIDIA put into CUDA when we made feature requests), or they were not reliably passing our numerical tests. Running OpenCL on NVIDIA GPUs was yielding less performance than OpenCL as well, so every year, the answer was "CUDA is the only viable HPC software ecosystem to invest long-term in".

CUDA has grown to become the standard of HPC, and this monopoly now reflects on NVIDIA's behaviour with its CUDA libraries. Their monopoly enables them to dictate the direction of their libraries (e.g. deprecations and new features, API breakages), and the scientific community must adapt to survive.

This year, like many others, we looked at AMD's available technologies, like ROCm, HSA and hip/SYCL, but our development team voted for restraint because we thought the API was not stable for the scale of investment we wanted to make for a cross-platform product. AMD has changed software direction many times in the last few years, and that is a risk that a company would need to absorb.

We asked AMD to provide us with a compute GPU and an official statement on what their recommended compute ecosystem is. What driver is recommended? What language should we write in? How do we write cross platform code? AMD provided us with an RX 5700XT, so we tried ROCm with it and ... it didn't work. We later found out, it was unsupported for use with ROCm. We asked them what use the compute GPU was if we couldn't run their first-party compute library on it, and it seems their vendor relations employee was not interested in compute as much as they were for gaming. We have yet to receive a decent enough answer from AMD on their compute stack and until we do, our team can not recommend the company to invest resources into AMD's ecosystem, even though customer requests for us to do so are plenty.

It seems to me that while AMD's hardware may be decent, their compute software division is not reliable enough to bank long-term investment in. They have a confused approach to compute. Contrast this with another newcomer on the HPC GPU turf: Intel.

Last year, Intel approached us to ask if we had heard about OneAPI. We said we had heard of it but hadn't looked at it in detail. Within a year of corresponding with Intel, we were able to have a sample of our product up and running. They helped port our CPU code (that already made use of MKL), to oneMKL, and provided us hardware and technical resources. The product is scheduled for release when Intel's HPC GPUs release, but it goes to show the difference in how serious each company takes their compute software division.

It hurts me that even after so many years of AMD producing compute GPUs, we are unable to develop code for them. Contrastingly, Intel just came within the past year and sent an engineer to help us port code and actively worked on our feature requests, resulting in us supporting their compute backend.

I just recently closed this year's examination into CUDA alternates for the company and with a heavy heart, I recommended not to invest in AMD again for the next year.

I agree with this, and am in much the same situation myself. To me, there’s a very important metric for a compute ecosystem, much more important than number of compute units, memory size, and certainly FPS: Size and quality of documentation. It’s clear that NVIDIA has invested heavily, over many years, in a huge library of documentation and examples that make it easy for developers to get started with CUDA programming or learn new CUDA features. By contrast, AMD hardly has anything. Maybe their technology is nearly as feature-rich as CUDA, but I have no way of knowing because their documentation isn’t nearly as comprehensive.

It sounds like your company also has more time/resources than mine to invest in looking into new compute ecosystems, so we probably won’t adopt ROCm ore OneAPI until those are much more stable or better supported. Hopefully, with some new AMD-based supercomputers coming up, the U.S. Department of Energy will push AMD to improve their documentation, or at least add some themselves.

It's laughable that they gave you a 5700xt when you asked for a compute oriented GPU. That's ridiculous, the 5700xt is notorious for not having Rocm support.

Do you guys do HPC stuff, or stuff that needs to run on individual computers? Because if the former, there's now the MI100.

I've been pretty disappointed in the lack of solid compute support on consumer GPUs since RDNA1. Although it's always been insisted that support is coming at some point, it's clearly not a high priority.

Yeah... My compute needs aren't quite as serious, but I feel cripped nevertheless with an AMD GPU.

I'd really like to be able to run deepdream, neural-style, big-sleep, styleGan (with pre-trained networks) etc etc, but they are all made on CUDA or have dependencies that require CUDA.

Some people have said that you can run this stuff with ROCm but I'm starting to feel that they're talking out of their asses, I have yet to see a working example. And it's a dependency hell when ROCm needs one version of some libraries and the programs I want to run need another version.

AMD needs to be able to run CUDA in one way or another, I don't care. I don't even care about the performance either at this point, when the ability is missing completely.
I'm just so frustrated about this. I'm almost ready to buy an overpriced Nvidia GPU... If there were any to buy anywhere. Other than the 3k€ 3090's.

I love reading posts from people who are smarter than me! Thank you for this rant. Very interesting.

AMD provided us with an RX 5700XT, so we tried ROCm with it and ... it didn't work.

that's hilarious seeing even some AMD employees didn't expect ROCm isn't supported on consumer RDNA

Within a year of corresponding with Intel, we were able to have a sample of our product up and running.

that sounds interesting. Difficulty-wise, how hard is oneAPI to learn and adapt? Around 2018 I was thinking to learn ROCm for university project but dropped the intention since the support for Vega 8 wasn't there and setting up ROCm is pretty tedious for my everyday machine. I don't plan to get nvidia gpu anytime soon but intel laptop with igp supporting oneAPI is pretty much easy and cheap to get.

I had a similar experience with Intel's OneAPI. I'm not important enough to get free support from them. But their API at least simply "works" in most cases.

AMD RocM is unsupported on Windows to start with, still has a few limitations regarding supported technologies, does not have any bindings at all (I couldn't even find unofficial ones but that may have a fluke) and isn't supported on my GPU (5700 XT) as well.

HIP seems to be a good replacement for CUDA, since it would use CUDA under the hood for Nvidia GPUs. However, the relationship with RocM is questionable (i.e. does HIP use RocM on AMD GPUs, ergo is not available for 5700 XT) and the support on windows right now is "maybe". The compiler architecture also seems needlessly complex and while it would add overhead, I'd love a vendor agnostic solution in the form of a HIP JIT rather than a compiler.

What segment of the hpc market are ya talking and when did you talk to amd? It looks like atm, amd's focus is on hyperscalers and datacenter hpc so recommending the 5700xt could mean that you're either not their focus market or their cdna work just ain't ready when you talked to them

Amd had very little against nvidia in compute before cdna and their focus is lookin to start small from the top to gain grounds before expanding the market. Amd has only hit big success last year with the zen2 processors, it'd be kinda unreal to expect thorough software support from them in just 1-2 years after the major breakthrough. Their desktop class architecture's more focused on gaming than compute atm

Completely agree. I work in HPC and have been following GPU compute very closely for years and it’s a shame that no good competitor to CUDA has emerged in the last 10 years.

On the AMD side it seems like there’s a new API every few years with OpenCL, Hip, SYCL, and ROCm. I know these are inter-related but it’s confusing for developers, especially scientists doing HPC. CUDA is easy to setup, easy to use, there’s lots of good libraries, good documentation, and support on all OSs on all their GPUs.

OpenCL has languished and it’s not as good to use (verbose code and lots of setup) in my experience. ROCm isn’t well supported (hardware-wise and software-wise). Maybe SYCL will be able to do something now the Intel is pushing it through their OneAPI.

You cannot compare these companies. nVidia and Intel is like a Supermarket with all that assistance personnel close to each section of goods when AMD is just a nice store with one cashier in it. Its just that AMD can do what they do performance wise is already a miracle. But stability and features AMD still nowhere near Intel or nVidia unfortunately.

Whoever decided to send you that 5700XT was a total moron. It's not a compute card, RDNA is for gaming, and AMD has split support for ROCm, Hip etc to their professional MI range. They should have sent you an MI50 as a bare minimum.

5700xt? now that's a Poor Volta moment.

Sounds about right...
It is a joke.

Why not use Vulkan as a compute library?

Seems to me the answer to this is "money" and having enough capable employees under contract (which also requires money). That's what starving a company for years does (guess which two anti-competitive companies...)

Intel has all the money they need and AMD would need to outsell them for years to come to even come close.

Do not waste your time. If you require GPU compute and you’re targeting linux or windows, just go with NVIDIA.

CUDA has had a 10+ year head start, and if you require BLAS or LAPACK you already have highly optimized library ecosystem for NVIDIA. And AMD is not likely to change any of that in the near future, they simply don’t have a good software strategy. It’s their Achilles heel and why they won’t overtake Intel in the data center or NVIDIA in the compute markets. Even if AMD has better CPUs or their GPUs can theoretically perform more compute. Sadly, all they could do until recently was to concentrate on the HW to keep the lights on, since they have been in rough financial shape until recently.

I never formally entered the ML space but I do remember almost a decade ago being a student in an informatics lab, where the lead scientist was a strong AMD enthusiast (mostly due to the superior Opteron's back then). We were all eagerly watching the new developments in using GPU's for compute, and we were all waiting to hear about developments in AMD and OpenCL. Kept waiting and waiting... pretty sure everyone just gave up after a while. I looked into it again recently and was amazed to see that the compute landscape for AMD GPU's has seemingly not progressed at all since those days almost a decade ago