How to know the GPU memory needed when training a detector network like faster R-CNN?

I have a GPU which only have 6G total memory. When training a faster R-CNN detector, although I have set the input size to 224*224*3, the miniBatchSize can only be set as 2. If I set miniBatchSize as 4 or 8 or larger, there are errors that out of memory on device and the data no longer exists on the device. Now I want to buy a new gpu which support me to train 1920*1080 picture with miniBatchSize set to be 64 or 128. But I don't know how to compute the memory and other paraneters the GPU needed. So how can I decide which GPU to choose?

Answer 1

[Mahesh Taparia]

Hi

I think 6GB GPU is enough for your code. Check if the code is running on CPU/ GPU. To run the code on GPU, set ExecutionEnvironment to 'gpu' , if you are using trainingOptions and trainNetwork function for training. You can refer this document for that. For custom training loop, you need to convert the array to gpuArray. For more information, refer this documentation of gpuArray.

[jingxue chen]

inputSize = [224 224 3];

preprocessedTrainingData = transform(trainingData, @(data)preprocessData(data,inputSize));
numAnchors = 5;
[anchorBoxes, meanIoU]  = estimateAnchorBoxes(preprocessedTrainingData,numAnchors)

featureExtractionNetwork = resnet50;
featureLayer = 'activation_40_relu';
numClasses = width(vehicleDataseti)-1;
lgraph = fasterRCNNLayers(inputSize,numClasses,anchorBoxes,featureExtractionNetwork,featureLayer);

%Data Augmentation 
augmentedTrainingData = transform(trainingData,@augmentData);
augmentedData = cell(4,1);
for k = 1:4
    data = read(augmentedTrainingData);
    augmentedData{k} = insertObjectAnnotation(data{1},'rectangle',data{2},data{3});
    reset(augmentedTrainingData);
end
figure
montage(augmentedData,'BorderSize',10)

%Preprocess Training Data 
trainingData = transform(augmentedTrainingData,@(data)preprocessData(data,inputSize));
validationData = transform(validationData,@(data)preprocessData(data,inputSize));

data = read(trainingData);
I = data{1};
bbox = data{2};
annotatedImage = insertObjectAnnotation(I,'rectangle',bbox,label);
annotatedImage = imresize(annotatedImage,2);
figure
imshow(annotatedImage)

%Train Faster R-CNN
options = trainingOptions('sgdm',...
    'MaxEpochs',4,...
    'MiniBatchSize',2,...
    'InitialLearnRate',1e-3,...
    'CheckpointPath',tempdir,...
    'ValidationData',validationData);
  % Train the Faster R-CNN detector.
    % * Adjust NegativeOverlapRange and PositiveOverlapRange to ensure
    %   that training samples tightly overlap with ground
    %   truth.                                                   
[detector, info] = trainFasterRCNNObjectDetector(trainingData,lgraph,options, ...
        'NegativeOverlapRange',[0 0.3], ...
        'PositiveOverlapRange',[0.6 1]);

Thanks a lot！

But when I am trying to train resnet50 + faster R-CNN on CPU use 1920*1080 images，set inputsize to 224*224 and miniBatchSize to 8，I found that sometimes memory used more than 24GB. （when training with such big miniBatchSize on GPU an error occurs as “The data no longer exists on the device.”）

When I set inputsize to 500*500 and miniBatchSize to 16，an error occurs that I am requesting 32*32*1024*13166（51.4GB）array. It' so big.

So I think maybe it's because there are some issue in the code（autually it's from the faster R-CNN example），or CPU and bad GPU needs more memory？（my GPU is GTX 980 Ti）

[Mahesh Taparia]

Hi

If you are using 1920*1080 size (HD) image, try to reduce its size and then start the training. Big images require more memory. Also, set the 'Execution Environment' to 'gpu' in the training option in your code, i.e

options = trainingOptions('sgdm',...
    'MaxEpochs',4,...
    'MiniBatchSize',2,...
    'InitialLearnRate',1e-3,...
    'CheckpointPath',tempdir,...
    'ValidationData',validationData,...
    'ExecutionEnvironment','gpu',...
    );

and check if the problem get resolve.