Convolutional LSTM (C-LSTM) in MATLAB

I'd like to train a convolutional neural network with an LSTM layer on the end of it. Similar to what was done in:

1. https://arxiv.org/pdf/1710.03804.pdf
2. https://arxiv.org/pdf/1612.01079.pdf

Is this possible?

Answer 1

[Shounak Mitra]

Hi Jake,

Unfortunately, we do not directly support C-LSTM. We are working on it and it should be available soon.

-- Shounak

[Seema Borase]

Hi Shonak,

Any updates on C-LSTM ?

[krishna Chauhan]

Ya same question is there any updat for same.

Also on attention layer?

[Girish Tiwari]

Hi Shounak,

Any update on C-LSTM in matlab 2021a?

[Zzz]

^

[Dieter Mayer]

Hello Shounak Mitra,

"Unfortunately, we do not directly support C-LSTM. We are working on it and it should be available soon."

After 4 years on working von C-LSTM, when do you thing, the use of convolutional LSTM networks will be available in Matlab?

Thanks in advance, best greetings,

Dieter

[David Willingham]

Hi Dieter,

Apologies for not updating this answers post sooner. This workflow is now supported. the following code will illustrated this:

% Load data
[XTrain,YTrain] = japaneseVowelsTrainData;

% Define layers
layers = [ sequenceInputLayer(12,'Normalization','none', 'MinLength', 9);
    convolution1dLayer(3, 16)
    batchNormalizationLayer()
    reluLayer()
    
    maxPooling1dLayer(2)
  
    convolution1dLayer(5, 32)
    batchNormalizationLayer()
    reluLayer() 
    
    averagePooling1dLayer(2)
    
    lstmLayer(100, 'OutputMode', 'last')

    fullyConnectedLayer(9)
    softmaxLayer() 
    classificationLayer()];

options = trainingOptions('adam', ...
    'MaxEpochs',10, ...
    'MiniBatchSize',27, ...
    'SequenceLength','longest');

% Train network
net = trainNetwork(XTrain,YTrain,layers,options);

[Dieter Mayer]

Hi David,

Thanks for your reply! Is this workflow shows a real convolution LSTM (LSTM carries out convolutional operations instead of matrix multiplication) and is not only implied to a input matrix, which is a result of a convolution net work applied before?

Sorry for asking that, I have to learn the syntax of using the deep learning toolbox, I am a beginner. The background is, that I will use such a Conv-LSTM to make precipitation forecasts for grids bases on precipitation radar inputs from several timesteps of the last minutes / hours as discussed in this paper publication

Answer 2

[Yi Wei]

Hi, can matlab support C-LSTM now?

[ytzhak goussha]

I have built something similar, not the same, by using fold-unfold option to incorporate CNN and LSTM in the same network.

[krishna Chauhan]

@ytzhak Could you plz eloborate in simple language.

Plz

[Girish Tiwari]

Hi Ytzhak,

Can you please explain how did you use sequenct fold-unfold layers to use CNN with LSTM?

[ytzhak goussha]

Hey,

Sorry I didn't follow this thread and didn't see the questions.

Here is a simplified C-LSTM network.

The input it a 4D image (height x width x channgle x time)

The input type is sqeuntial.

When you need to put CNN segments, you simply unfold->CNN->Fold->flatten and feed to LSTM layer.

[Ioana Cretu]

Hi! When I try to train the model I have this error:

Error using trainNetwork (line 170)
Invalid network.

Caused by:
    Layer 'fold': Unconnected output. Each layer output must be connected to the input of another layer.
    Detected unconnected outputs:
        output 'miniBatchSize'
    Layer 'unfold': Unconnected input. Each layer input must be connected to the output of another layer.
    

I connected the layers using this:

    lgraph = layerGraph(Layers);
lgraph = connectLayers(lgraph,'fold/miniBatchSize','unfold/miniBatchSize');

What do you think the cause is?

Answer 3

[Chen]

Please refer to this excellent example in:

Time Series Forecasting Using Hybrid CNN - RNN - File Exchange - MATLAB Central (mathworks.com)

It is possible to train the hybrid together.

Answer 4

[Jonathan]

inputSize = [28 28 1];
filterSize = 5;
numFilters = 20;
numHiddenUnits = 200;
numClasses = 10;
layers = [ ...
    sequenceInputLayer(inputSize,'Name','input')
    sequenceFoldingLayer('Name','fold')
    
    convolution2dLayer(filterSize,numFilters,'Name','conv')
    batchNormalizationLayer('Name','bn')
    reluLayer('Name','relu')
    
    sequenceUnfoldingLayer('Name','unfold')
    flattenLayer('Name','flatten')
    lstmLayer(numHiddenUnits,'OutputMode','last','Name','lstm')
    fullyConnectedLayer(numClasses, 'Name','fc')
    softmaxLayer('Name','softmax')
    classificationLayer('Name','classification')];
lgraph = layerGraph(layers);
lgraph = connectLayers(lgraph,'fold/miniBatchSize','unfold/miniBatchSize');

https://www.mathworks.com/help/deeplearning/ug/long-short-term-memory-networks.html

Answer 5

[David Willingham]

Updating this answer. This workflow has been supported since R2021. The following example illustrates how to combin CNN's with LSTM layers:

% Load data
[XTrain,YTrain] = japaneseVowelsTrainData;

% Define layers
layers = [ sequenceInputLayer(12,'Normalization','none', 'MinLength', 9);
    convolution1dLayer(3, 16)
    batchNormalizationLayer()
    reluLayer()
    
    maxPooling1dLayer(2)
  
    convolution1dLayer(5, 32)
    batchNormalizationLayer()
    reluLayer() 
    
    averagePooling1dLayer(2)
    
    lstmLayer(100, 'OutputMode', 'last')

    fullyConnectedLayer(9)
    softmaxLayer() 
    classificationLayer()];

options = trainingOptions('adam', ...
    'MaxEpochs',10, ...
    'MiniBatchSize',27, ...
    'SequenceLength','longest');

% Train network
net = trainNetwork(XTrain,YTrain,layers,options);