Multi Input Multi Output Machine Learning model for varying length sequence data.

Dear reader,

I have been trying to create a Multi Input Multi Output Machine Learning model in Matlab for sequence data which is varying in length. The source of inspiration for my solution has been the following two examples:

• https://nl.mathworks.com/help/deeplearning/ug/time-series-forecasting-using-deep-learning.html referred to as sequence.
• https://nl.mathworks.com/help/deeplearning/ug/train-network-with-multiple-outputs.html referred to as MIMO.

After not getting it to work in my main code, I created (together with chatgpt) this example code to test certain variations.

close all, clear all, clc

% Step 1: Prepare sequence data (input and output) with varying lengths

% Data in [Features x Time] format

input1 = {rand(3,10)'; rand(3,8)'; rand(3,12)'}; % Input 1: 3 features

input2 = {rand(5,10)'; rand(5,8)'; rand(5,12)'}; % Input 2: 5 features

output1 = {rand(2,10)'; rand(2,8)'; rand(2,12)'}; % Output 1: 2 features

output2 = {rand(1,10)'; rand(1,8)'; rand(1,12)'}; % Output 2: 1 feature

% Combine input data into cell array

inputs = {input1, input2};

outputs = {output1, output2};

% Step 2: Define Network Architecture with multiple inputs and multiple outputs

% Input layers for each input

inputLayer1 = sequenceInputLayer(3,'Name','input_1'); % 3 features for input 1

inputLayer2 = sequenceInputLayer(5,'Name','input_2'); % 5 features for input 2

% LSTM layers for input processing

lstmLayer1 = lstmLayer(10,'Name','lstm_1'); % Processes input 1

lstmLayer2 = lstmLayer(20,'Name','lstm_2'); % Processes input 2

% Concatenation layers (concatenating along channels dimension)

concatLayer1 = concatenationLayer(1,2,'Name','concatLayer1');

concatLayer2 = concatenationLayer(1,2,'Name','concatLayer2');

% Fully connected layers for the outputs

fc1 = fullyConnectedLayer(2,'Name','fc_1'); % Output 1: 2 features

fc2 = fullyConnectedLayer(1,'Name','fc_2'); % Output 2: 1 feature

% Create layer graph with branching for multiple inputs and outputs

layers = layerGraph();

% Add and connect layers for input 1

layers = addLayers(layers, inputLayer1);

layers = addLayers(layers, lstmLayer1);

layers = addLayers(layers, concatLayer1);

layers = addLayers(layers, fc1);

% Add and connect layers for input 2

layers = addLayers(layers, inputLayer2);

layers = addLayers(layers, lstmLayer2);

layers = addLayers(layers, concatLayer2);

layers = addLayers(layers, fc2);

% Connect network branches

layers = connectLayers(layers, 'input_1', 'lstm_1');

layers = connectLayers(layers, 'input_2', 'lstm_2');

layers = connectLayers(layers, 'lstm_1', 'concatLayer1/in1');

layers = connectLayers(layers, 'lstm_2', 'concatLayer1/in2');

layers = connectLayers(layers, 'lstm_2', 'concatLayer2/in1');

layers = connectLayers(layers, 'lstm_1', 'concatLayer2/in2');

layers = connectLayers(layers, 'concatLayer1', 'fc_1');

layers = connectLayers(layers, 'concatLayer2', 'fc_2');

% Convert to dlnetwork object

dlnet = dlnetwork(layers);

% Step 3: Define custom loss function for use with trainnet

function loss = modelLoss(dlnet, dlX, dlY)

% Extract individual inputs and outputs

dlX1 = dlX{1};

dlX2 = dlX{2};

dlY1 = dlY{1};

dlY2 = dlY{2};

% Forward pass through the network

dlYPred1 = forward(dlnet, dlX1, 'input_1');

dlYPred2 = forward(dlnet, dlX2, 'input_2');

% Compute loss for both outputs

loss1 = mse(dlYPred1, dlY1);

loss2 = mse(dlYPred2, dlY2);

loss = loss1 + loss2;

end

% Step 4: Create a datastore for training

inputDatastore = arrayDatastore(inputs, 'IterationDimension', 1);

outputDatastore = arrayDatastore(outputs, 'IterationDimension', 1);

% Combine the input and output datastores

ds = combine(inputDatastore, outputDatastore);

% Training options

options = trainingOptions('adam', ...

'MaxEpochs', 100, ...

'MiniBatchSize', 3, ...

'Plots', 'training-progress');

% Display the network structure

figure

plot(layers);

% Step 5: Train the network using trainnet

trainedNet = trainnet(ds, dlnet, @modelLoss, options);

Sadly, the following error occurs:

Error using trainnet (line 46)

Error forming mini-batch for network input "input_1". Data interpreted with format "TCB". To specify a different format, use the InputDataFormats option.

Error in test_chatgpt (line 97)

trainedNet = trainnet(ds, dlnet, @modelLoss, options);

Caused by:

Input sequences must be numeric or categorical arrays.

The error says that the data is interpreted as TCB, the sequence example I think also uses TCB after testing several options (default is 'auto'). Therefor the error might not be the actual error.

So I'm wondering, are there options to solve this? As far as I understand does the data handler not understand how to read the data eventhough both examples show the possibility for it individually.

One solution would be creating a 3D array with padded data. But I would not prefer to use padded data since that could have effect on the model since different lengths resemble different speeds in my scenario and that could influence the model.

Another solution would be to make all the samples the same length without padding and adding them to a 3D array. However that would reduce the amount of data for the model to train, but does not introduce an indirect variable to the model.

Thanks in advance for your time.

Answer 1

[Gayathri]

Hi @Corné Dijkstra,

I understand that you are facing error while trying to train a “Multiple Input Multiple Output” Neural Network.

To solve the error “Input sequences must be numeric or categorical arrays.”, we can first convert the inputs and outputs from cell array to numeric arrays before converting it to an “arrayDatastore” object. And as the network expects two inputs and two outputs, it is advised to combine the “arrayDatastore” objects separately for each input and output. Please refer to the below codes for your reference.

convertToNumeric = @(cellArray) cellfun(@(x) double(x), cellArray, 'UniformOutput', false);

% Convert cell arrays to numeric arrays
inputnumeric1 = cell2mat(convertToNumeric(input1));
inputnumeric2 = cell2mat(convertToNumeric(input2));
outputnumeric1 = cell2mat(convertToNumeric(output1));
outputnumeric2 = cell2mat(convertToNumeric(output2));
% Create datastores for each input and output
inputDatastore1 = arrayDatastore(inputnumeric1, 'IterationDimension', 1);
inputDatastore2 = arrayDatastore(inputnumeric2, 'IterationDimension', 1);
outputDatastore1 = arrayDatastore(outputnumeric1, 'IterationDimension', 1);
outputDatastore2 = arrayDatastore(outputnumeric2, 'IterationDimension', 1);
% Combine the datastores into a single datastore
ds = combine(inputDatastore1, inputDatastore2, outputDatastore1, outputDatastore2);

Next, we can make one additional modification to the code provided. I understand that, you want to use “mse” as loss function for both the set of data. Therefore, please mention the loss function in the below format and then pass it to the “trainnet” function.

lossFcn = @(Y1,Y2,T1,T2) mse(Y1,T1) + mse(Y2,T2);
% Step 5: Train the network using trainnet
trainedNet = trainnet(ds, dlnet, lossFcn, options);

For more information on training a “Multiple-Input and Multiple-Output Networks”, please refer to the below mentioned link.

https://in.mathworks.com/help/releases/R2024a/deeplearning/ug/multiple-input-and-multiple-output-networks.html

Hope you find this information helpful.

[Corné Dijkstra]

Dear Gayathri,

Thanks for your quick reply. Your solution does make the ML run, however the purpose of the ML would be to learn from different subjects and combining the data into 1 set without separation would not be the desired outcome.

Maybe a general question to Mathworks, how are datastores supposed to function? I compare them to pointes in C++, pointing to the data to be used. Since in the sequence example a cell array does function for trainnet, I cannot fathom why a cell array would not function in combination with a datastore. Evenmore, why does for a MIMO system the input and output be placed together in the datastore? It makes it only more difficult for the code to determine what is supposed to be what. Since apart from size I haven't found another determining factor how it differeniates which data should be connected where in the ML.