Reshape error in trainNetwork with fileDatastore

Hi,

I try to train NN with fileDatastore, where input is 1x21x1x100 in each mat file and output is 1x1x21x100 in each mat file.

My layers are:

inputsize=21;

Layers = [

imageInputLayer([1 inputsize 1],'Normalization','none')

fullyConnectedLayer(512)

reluLayer

fullyConnectedLayer(64)

reluLayer

fullyConnectedLayer(32)

reluLayer

fullyConnectedLayer(inputsize)

regressionLayer];

and the way I code fileDatastore is:

input = fileDatastore('Dataset_input','ReadFcn',@load);

inputDatat = transform(input,@(data) rearrange_datastore_input(data));

output = fileDatastore('Dataset_output','ReadFcn',@load);

outputDatat = transform(output,@(data) rearrange_datastore_output(data));

trainData=combine(inputDatat,outputDatat);

function image = rearrange_datastore_input(data)

image=data.input;

image= {image};

end

function image = rearrange_datastore_output(data)

image=data.output;

image= {image};

end

However, I keep getting errors as:

Error using trainNetwork (line 170)

Number of elements must not change. Use [] as one of the size inputs to automatically calculate the

appropriate size for that dimension.

Caused by:

Error using reshape

Number of elements must not change. Use [] as one of the size inputs to automatically calculate the

appropriate size for that dimension.

I tried to debug a bit and find the problem is in singleInputSingleOutputCollateFcns.m file. Since my output is 1x1x21x100, matlab reshapes output for regression, where error occurs. When reshape, function reshapedCell = cellfun(@(x)reshape(x,outputSizes), cellData(:,col), 'UniformOutput', false); is called, where outputSizes=1,1,21; and element in cellData is of size 1x1x21x100, and thus reshape error occurs.

Is this problem caused by my output form? How should I set the dimension of output?

[Jeremy Hughes]

My guess is that "1x21x1x100 in each mat file and output is 1x1x21x100" being different, there's something unexpected going on.

Does the same thing occur if you add a reshape to 21x100 in your rearrange_datastore_output function?

(PS, if you use the code format option in your post, it will be easier to understand your code.)

[Haoqing Li]

Thanks, I think I find the reason. In filedatastore, the mat file can only be one sample for input. In my input, by 1x21x1x100, there are in fact 100 input samples. Therefore, I should save 1x21x1 in one mat file. Is that right?

Answer 1

[Aylin]

> Therefore, I should save 1x21x1 in one mat file. Is that right?

Hi Haoqing, yes, separating your observations into different MAT files will help trainNetwork understand that they are different observations.

Refer to the Datastores for Deep Learning doc page for some guidelines on how to organize your data when it is input to trainNetwork.

Based on that doc page, if you prefer to maintain your data in a single MAT file, then you could also use num2cell to convert a 1x21x1x100 matrix into a 1x100 cell using something like:

function image = rearrange_datastore_input(data)
    image = data.input;
    image = num2cell(image, 1:3); % Wrap 1x21x1x100 data in 1x1x1x100 cell
    image = image(:); % Reshape 1x1x1x100 cell to 1x100 cell
end

I hope this helps!