What happen to the CUDA cache mem?

Hello there. I am a newbie with the GPU computing with Matlab, so apologize if the question sounds silly. I am trying to optimise some computation I am doing with the GPU. I believe it is well configured. I am doing some testing to understand how the GPU reacts to different commands and choose the best programming strategy. I have incurred in the following thing. I would appreciate some elucidation about the mechanism by which this feature happens. I am running a Geforce GTX 1080 ti. I do the following:

A = rand([100 100 100 100 10],'single','gpuArray')
tic,permute(A,[3 2 1 5 4]),wait(M.SelectedDevice()),toc

(trying to see how long does it take to permute a matrix)

now, if I ask the parallel.gpu.GPUDeviceManager.instance.SelectedDevice().AvailableMemory (read the available memory), then I can run a permute again. However, if I run two consecutive permute, I get the following

Error using gpuArray/permute Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'.

WHY?

[Walter Roberson]

Have you tried calling gather() after the permute?

[fpexp]

nope, in fact the result is not stored anywhere. I would have expected the RAM area to be released immediately

Answer 1

[Joss Knight]

The result is stored as the variable ans, which means you have less memory the second time round.

[giovanni esposito]

Hello, I am using four GPU in an spmd process. the spmd is inside a for loop.

1) gpu memory usage increase with loops, so I am forced to reset device every RefreshGPU times. Although gpu reset, most part of memory is not freed

2) the original code crash after a large number of loops returning "An invalid FFT Plan on GPU"

any help about?

thank you

here is a simplified part of code

RefreshGPU = 100;NW =  gpuDeviceCount;
for ii = 1:NW
    GpuD = gpuDevice(ii);
    GpuD.reset;    
    GpuD.wait
end
clear GpuD
nw = 1:NW;
poolobj = gcp('nocreate'); % If no pool, do not create new one.
if isempty(poolobj)
    parpool('local',NW);
else
    delete(gcp);

      parpool('local',NW);
  end
  a=rand(NW,1e4);
  Nloop = 1e5;
  for kk=1:Nloop  
      if not(mod(kk,100))       
          for ii = 1:NW
              GpuD = gpuDevice(ii);
              GpuD.reset;
              GpuD.wait
          end
      end
      clear GpuD    
      spmd
          ag=gpuArray(a(labindex,:));
          b=fft(ag,nextpow2(size(ag,2)));
      end
  end

[Joss Knight]

This code does not reset GPUs on the workers, it does it on the client. You haven't done anything on the workers until you reach the SPMD block.

Resetting the device is a very heavy hammer. You should just clear variables from the workspace, or better, use functions so that temporary variables are automatically cleared when the function exits.

[giovanni esposito]

hence, for example this code shall free all gpus memory ad the end of each loop, correct ? I try to do this but memory is still busy at the end of each loop.

clear all
RefreshGPU = 100;
NW =  gpuDeviceCount;

nw = 1:NW;
poolobj = gcp('nocreate'); % If no pool, do not create new one.
if isempty(poolobj)
    ParObj = parpool('local',NW);
else
    delete(gcp);

      ParObj = parpool('local',NW);
  end

a=rand(NW,1e5);
Nloop = 1e5;
for kk=1:Nloop    
    spmd
        b = somefunction(a(labindex,:)); % this function do something on GPUs
    end    
    clear b
end

[Joss Knight]

No, you are calling clear b on the client. You need to do it inside the SPMD block.

Answer 2

[Jeffrey Daniels]

FYI - For anyone else having similar problems, I get similar errors when I run too many workers. The GPU is being shared by each of the CPU workers and if you have too large or too many GPU matricies you will run out of memory on the GPU. One solution is to open the Cluster Profile Manager from the Parallel menu and reduce the number of workers in your Cluster Profile.