Trouble using one of my GPUs

I am using a computer with Ubuntu 20.04. I want to use a MATLAB software toolkit that uses mex files to do medical image reconstruction. The image reconstructions with this toolkit work on one of my GPUs (GeForce GTX TITAN X) but when I try using the other GPU (TITAN RTX) for this, I get the following Cuda failed error:

no kernel image is available for execution on the device

Please clear any outstanding memory allocations and reset the device(s) in use.

Here is the output of nvidia-smi as a .png file

I'd be happy to provide more information. I have not seen similar errors when using pytorch on this computer.

What should I try to resolve this error so that I can run the image reconstructions on the TITAN RTX?

Thanks,

Rohan

Answer 1

[Dinesh]

Hi Rohan.

The error "no kernel image is available for execution on the device" typically indicates that the CUDA code has been compiled for a specific Compute Capability, and the GPU you're trying to use has a different Compute Capability not supported by the compiled code.

The GeForce GTX TITAN X has a Compute Capability (CC) of 5.2, while the TITAN RTX has a Compute Capability of 7.5.

The following link might help you:

https://github.com/pytorch/pytorch/issues/31285

[Rohan Nadkarni]

Thanks.

Like you said, the Cuda code was compiled for capability of 7.5.

I eventually figured out that MATLAB's gpuDevice and nvidia-smi were not indexing the GPU's in the same order (gpuDevice(1) was GPU 1 in nvidia-smi rather than GPU 0 in nvidia-smi).

So it is actually the GeForce GTX TITAN X that gave me the error due to insufficient compute capability, but the TITAN RTX could execute the Cuda code without errors.

Answer 2

[Joss Knight]

Hi Rohan. You may have to consult the documentation for your toolkit. The Titan RTX is a Turing card and may post-date the most recent build of your toolkit. Really the toolkit authors should have built a forward compatible bytecode version but perhaps they expect you to just upgrade to the latest version.