Training nework: Combined network is not partitionable but each datastore is partitionable

I was training an regression network which basically maps noisy image to noise. The input and output are both 384x384x8x2.

The input data is in folder "Train_Noisy", the target data is in folder "Train_Noise", each file is in a mat file.

addpath('Train_Noisy')
addpath('Train_Noise')

Input_Data=fileDatastore(fullfile('Train_Noisy'),'ReadFcn',@Load_Noisy,'FileExtensions','.mat');
Target_Data=fileDatastore(fullfile('Train_Noise'),'ReadFcn',@Load_Noise,'FileExtensions','.mat');

Train_Data=combine(Input_Data,Target_Data);


function Noisy = Load_Noisy(file)
File = load(file);
Noisy = File.Noisy;
end

function Noise = Load_Noise(file)
File = load(file);
Noise = File.Noise;
end

I have two different GPU installed, RTX 2080Ti and RTX 2080.

The training options is

options = trainingOptions('adam', ...
    'ExecutionEnvironment','multi-gpu',...
    'LearnRateSchedule','piecewise', ...
    'InitialLearnRate',1e-3,...
    'LearnRateDropFactor',0.9, ...
    'LearnRateDropPeriod',1, ...
    'MaxEpochs',100, ...
    'MiniBatchSize',1, ...
    'VerboseFrequency',10,...
    'Plots','training-progress')

However, Matlab gives me the error

The input datastore is not Partitionable and does not support parallel operations.

Then I go back and check the inputData, targetData and the combined datastore trainData

>> isPartitionable(Target_Data)
isPartitionable(Input_Data)
isPartitionable(Train_Data)

ans =

  logical

   1


ans =

  logical

   1


ans =

  logical

   0

Thus each datastore is partitionable but the combined is not.

From the isPartitionable, CombinedDatastore is partitionable if all underlying datastores have a subset method or are transformations/combinations of datastores that have subset methods.

I do not know how to add the subset method for each datastore and for the CombinedDatastore.

Answer 1

[Aylin]

Hello Shen Zhao,

This is a known limitation in the FileDatastore + combine workflow.

Even though FileDatastore by itself is partitionable, combining two FileDatastores is currently not defined as partitionable. We are working on ways to remove this limitation in future releases.

One possible workaround for this is to directly import each pair of MAT files in a single FileDatastore. You would need to have some kind of mapping between the filenames used in the Train_Noisy/ and Train_Noise/ datasets for this to work:

addpath('Train_Noisy')
addpath('Train_Noise')

Train_Data = fileDatastore(fullfile('Train_Noisy'), 'ReadFcn', @Load_All, 'FileExtensions', '.mat');

function Data = Load_All(noisy_filename)
    File = load(noisy_filename);
    Noisy = File.Noisy;

    noise_filename = Make_Noise_Filename(noisy_filename);
    File = load(noise_filename);
    Noise = File.Noise;

    Data = [Noisy Noise];
end

You'd also need to define another function here called Make_Noise_Filename that converts filenames from the first dataset to the second dataset. This sidesteps the limitation in FileDatastore + combine for now.

I hope this helps!

Rylan

[Shen Zhao]

Thank you Rylan.

I have also made it work by replacing fileDatastore with imageDatastore, nothing else chaged.

[Jari Manni]

Hi @Rylan Dmello,

is there any updates with this issue. I have come across similar problem when applying custom data augmentation to pixelLabelImageDatastore with transform(...) function, which returns a TransformedDatastore. It somehow becomes not partitionable afterwards.