How to prepare the training data for neural net with concatenationLayer, which accepts the combination of sequence inputs and normal inputs?

I looked into the example code for Vector-Sequence and Feature Classification Network in MATLAB example: Example Deep Learning Networks Architectures. The URL is here https://www.mathworks.com/help/deeplearning/ug/example-deep-learning-networks.html

In the visualization of the neural net structure, there are two input sources and a concatenationLayer in the middle, as below

On the top left is the sequence input, suppose I have two channels in the sequence input with the same fixed length (100). And on the top right for the input layer, suppose I would like to feed a vector of size 5. And I have 1000 training points. The output is a scaler. In other words, each output is associated with a sequence input (2X100) and a vector of size 5.

Given the trainNetwork(XTrain,YTrain,layers,options), with layers as the neural net layer, in what order should I arrange the data in XTrain to make sure the sequence input and the vector input are correctly identified and used for training? I have done some feedfoward neural net training and I know that the number of column means the number of training data, given XTrain,YTrain are 2D matrices.

But things will get complex when the sequence input has more than one channels, since from other posts, I notice that the sequence training data are arranged in cell array of 2D matrices, e.g., see MATLAB Example: Sequence Classification Using Deep Learning. The URL is here: https://www.mathworks.com/help/deeplearning/ug/classify-sequence-data-using-lstm-networks.html.

So when I have both sequence input and the normal vector input, what form will my XTrain be?

I clicked the Example tab and could not find a relevant usage of training this kind of neural net. So I wrote a post here.

Answers and relevant links are greatly appreciate it, :)!

[Liangwu Yan]

I think I found a relevant MATLAB example (Train Network on Image and Feature Data) which could help me. The URL is here: https://www.mathworks.com/help/deeplearning/ug/train-network-on-image-and-feature-data.html

In the example, the training data are converted into datastore Type via arrayDatastore and then combined into dsTrain, as seen in the picture below

Seems like the sequence of the combined data is the same as the input required by the neural net, as seen below

dsTrain = combine(dsX1Train,dsX2Train,dsTTrain);

dsX1Train(ImageInput), dsX2Train(rotation angle), dsTTrain(output).

Am I correct?

However, an answer from an experienced user or Mathworker would help a lot, :D.

Answer 1

[Ben]

You are right that to use trainNetwork with a network that has multiple inputs you will need to use a datastore. There is documentation for this here https://uk.mathworks.com/help/deeplearning/ug/multiple-input-and-multiple-output-networks.html which also links to the example you have found for image and feature data. Sequence data can be used as an additional input so long as the datastore reads it out in our expected format - a cell array where each entry is one sequence with the sequence dimension last.

You can also use dlnetwork and a custom training loop, in which case you don't necessarily need datastores as you can directly call forward(net, x1, x2) in your training loop, where net is a dlnetwork, and x1,x2 are the two input data. However this may require more effort if you haven't used custom training loops before.