Out of memory issue on evaluating CNNs

The message "Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'." appears when I try to evaluate my trained CNN. I'm using a GeForce 1060M GTX 6GB RAM.

Here's a piece of my code:

testData = load('testROI.mat');
[test_imds, test_pxds] = pixelLabelTrainingData(testData.gTruth);
testDataSet = pixelLabelImageDatastore(test_imds, test_pxds);
unetPxdsTruth = testDataSet.PixelLabelData;
unetpxdsResults = semanticseg(test_imds, unet); % error is caused by this line
unetMetrics = evaluateSemanticSegmentation(unetpxdsResults, unetPxdsTruth);

The command gpuDevice() shows the results below:

CUDADevice with properties:

                      Name: 'GeForce GTX 1060'
                     Index: 1
         ComputeCapability: '6.1'
            SupportsDouble: 1
             DriverVersion: 9.2000
            ToolkitVersion: 9.1000
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 6.4425e+09
           AvailableMemory: 5.0524e+09
       MultiprocessorCount: 10
              ClockRateKHz: 1670500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

As you can see, there are more than 5GB of free memoy but, for some reason I don't understand, the out of memory problem happens. The curious thing is it doesn't happen with 500 images the training stage, but happens with 100 images in the test evaluating stage. It's important to emphasize that this evaluation atempt uses a pretrained CNN that I created in another moment, so the training data is not in the GPU memory while doing this.

Does anyone please knows what might be going on?

Answer 1

[Andrea Picciau]

The problem is that your GPU's 6GB memory is not enough to execute the semantic segmentation with the default settings. Reducing the mini-batch size from the default 126 should be enough in your case. Try changing the problematic line to the following:

unetpxdsResults = semanticseg(test_imds, unet, 'MiniBatchSize', 4);

You can try increasing that 4 to a larger value, but it wouldn't surprise me if 8 was the maximum your GPU could get to with your GPU.

You should also have a look at semanticseg's doc page and the name-value pairs in particular.

A last note on gpuDevice: when you get the out-of-memory error, MATLAB doesn't allocate the data. This is approximately what happens:

• At rest, something between a few hundred MBs to a GB is allocated on your GPU memory. This is the space occupied by the CUDA libraries.
• When you are running sematicseg with the default settings, some of the data MATLAB needs to allocate is way larger than your GPU's remaining 5GB memory.
• MATLAB asks CUDA to allocate that data,
• CUDA gives an error, saying that your GPU's memory is not large enough,
• MATLAB informs you with the error message and doesn't allocate anything,
• When you check with gpuDevice, you see 5GB are free.

[Andrea Picciau]

Also, I'm not sure what MATLAB version you're using but the memory use of semanticseg has been largely reduced starting from R2018a update 1.

[ioannisbaptista]

Hello Andrea. First of all, thank you so much for answering me.

I'm glad to tell that this solution does work. I found it out just before you answer me.

The application was getting out of memory indeed: after starting, GPU peaks quickly reached limit before exception occured. I set the MiniBatchSize to 32, like this:

unetpxdsResults = semanticseg(test_imds, unet, 'MiniBatchSize', 32);

so it worked. Setting to 64 (assuming it has to be a 2^k value), GPU couldn't handle it as well.

About the notes on gpuDevice: the application starts with a few hundred MB used on the GPU, indeed. But after the error, the used memory remains 1.5~2.0 GB. So, it looks like something is allocated a bit, I just don'to know what.

Seizing the opportunity, I posted just part of my code, but the truth is I evaluate three CNNs (U-Net, SegNet and Dilated Convolutions Net. There are three command blocks like that in the same script (except for the data loading, this is done just once). Do you know if after evaluating a CNN the used memory is released for the next one?

[Andrea Picciau]

I'm happy to hear you solved the problem!

You're right, the thing is not actually as simple as I put it in my reply (that's why I said "approximately"!). In reality, MATLAB is doing many different GPU memory optimisations at the same time: it loads some CUDA libraries only when it needs to, it keeps a bit of GPU memory ready to speed-up allocations...that's why the GPU memory is a bit different before and after. In practice, you shouldn't be concerned. When you need your GPU memory, MATLAB is going to allocate as much of it as you want because it doesn't use any GPU memory for itself.

To ensure you make full use of your GPU memory, the main trick is not to use your GPU to drive a display at the same time as doing computations.

[ioannisbaptista]

Got it, Andrea.

Thank you so much! Regards.

Answer 2

[silver tena]

Hello Andrea and ioannisbaptista.

I have a problem when I am training the program for image classification. The error message is "out of memory". My GPU specification bellow.

Name: 'GeForce GTX 1650'

Index: 1

ComputeCapability: '7.5'

SupportsDouble: 1

DriverVersion: 11.1000

ToolkitVersion: 11

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 4.2950e+09

AvailableMemory: 3.3381e+09

MultiprocessorCount: 14

ClockRateKHz: 1710000

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 1

CanMapHostMemory: 1

DeviceSupported: 1

DeviceAvailable: 1

DeviceSelected: 1

I try to training for alexnet, it's not problem. When I training for VGG16, the problem occured as "out of memory".

This part is where the error occurs. I am set teh minibatch size=5. epoch =10. Amount of pictures are 300 only.

featuresTrain = activations(netTransfer,augimdsTrain,layer,'OutputAs','rows');

I need your help.

I still have to train also the Resnet 110 and iceptionV3 architecture. I need the help of both of you. God bless all fo you.