An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR​_ILLEGAL_A​DDRESS

I try to execute the following test to use GPU computing in Matlab2016a but always get an error about illegal memory access, which I don't know how to resolve.

Driver

N = 6;
M = magic(N);
G = gpuArray(M);
Error using gpuArray
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

I checked my GPU device and get the following report which seems fine

gpuDevice(1)

ans = 

    CUDADevice with properties:

                        Name: 'GeForce GTX 1080'
                       Index: 1
           ComputeCapability: '6.1'
              SupportsDouble: 1
               DriverVersion: 8
              ToolkitVersion: 7.5000
          MaxThreadsPerBlock: 1024
            MaxShmemPerBlock: 49152
          MaxThreadBlockSize: [1024 1024 64]
                 MaxGridSize: [2.1475e+09 65535 65535]
                   SIMDWidth: 32
                 TotalMemory: 8.5899e+09
             AvailableMemory: 8.0519e+09
         MultiprocessorCount: 20
                ClockRateKHz: 1809500
                 ComputeMode: 'Default'
        GPUOverlapsTransfers: 1
      KernelExecutionTimeout: 1
            CanMapHostMemory: 1
             DeviceSupported: 1
              DeviceSelected: 1

Also this seems to work fine:

mexcuda mexGPUExample.cu
Building with 'NVIDIA CUDA Compiler'.
MEX completed successfully.

I try to use the GPU computing for the first time on a Windows7, Visual Studio Communit 2013, CUDA 7.5. I also set the system variable for the CUDA_Cache_MAXSIZE but I am unsure what is wrong with the access to the device. Does anyone have experience with such an issue?

Answer 1

[Joss Knight]

CUDA 7.5 has significant issues with Pascal cards.

https://www.mathworks.com/matlabcentral/answers/309235-can-i-use-my-nvidia-pascal-architecture-gpu-with-matlab-for-gpu-computing

The first port of call is to make sure you have the most up-to-date drivers for your particular card, by downloading them from the NVIDIA website. If this doesn't help, tech support may be able to help you work out how to work around the issue for your particular use case, until MATLAB R2017a comes out (which uses CUDA 8.0 and has full support for Pascal).

Most things work, but there are some significant issues, particularly on Windows 7 (CONV2, FILTER, INTERP3).

Answer 2

[SFSpikes]

Hi,

Thanks for your feedback. I have the most up-to-date drivers for my NVDIDIA GEForce GTX1080 card and I set the MAX_SIZE_CACHE as described. Yet I still get the same error message and will need to use FILTER. Any other solutions?

[Joss Knight]

I've got to say, your reproduction is extremely unusual. I myself can successfully run this code on Windows 7 on a GTX 1080 in MATLAB R2016a. I would say that probably there's something wrong with your setup, because you've got a failure to just copy data to the card, whereas the problems with Pascal cards are caused by errors in the code generation for MATLAB kernels.

You might want to confirm that you can compile and run some of the CUDA samples that come with the toolkit.

[SFSpikes]

I tried to use different function in the meantime and the error seems to specifically occur when I try to use the filter function.

For example N = 6; M = magic(N); G = gpuArray(M)

G =

    35     1     6    26    19    24
     3    32     7    21    23    25
    31     9     2    22    27    20
     8    28    33    17    10    15
    30     5    34    12    14    16
     4    36    29    13    18    11

However when I call the filter function the error Warning: An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_ILLEGAL_ADDRESS. After this error occurs even the gpuArray(M) call will throw the same error and I have to close Matlab to reset the GPU.

I also did a deviceQuery and the output seemed fine. I will test some samples now and send you an update. So far I tried the bandWidthTest wich ran fine the result was a Pass. I also tried the cuSolverRF which also ran fine it seems. Thanks for your help!

[Joss Knight]

Okay. Well, this is because FILTER isn't working properly on Pascal cards. If you have access to the R2017a prerelease I suggest to do that. Otherwise, if your filter is simple then you can probably reimplement it using FFT/IFFT.

Once you have run a kernel that errors with an illegal address access like this, you need to reset the card or every use of the GPU will return the same error.

[SFSpikes]

Ok get it, thanks a lot. Unfortunately I have no access to a prerelease of 2017a

[Nadeem Ansari]

Is your problem is resolved?

I have same issue.

Please suggest

[Walter Roberson]

@Nadeem Ansari

Which MATLAB release are you using? What kind of GPU do you have? What CUDA driver version do you have installed?

[Nadeem Ansari]

MATLAB 2021

[Walter Roberson]

It looks to me as if that driver version is correct for that device, and that that class of device should be supported by R2021a.

I think you are going to need to open a support case with Mathworks.