How to use the LSTM network to forecast the image?

112 views (last 30 days)
Ning Liu
Ning Liu on 22 May 2021
Commented: Ning Liu on 19 Sep 2021
Hi, I have image time series datasets and each image size is 785*785*3, the time series length is 400. Now I want to establish a LSTM network to fit , is the image at time t and is the image at time t-1. I use the previrous 350 images to prepare the train data and the last 50 images to test the forecast results. I construct a simple LSTM network as follows,
layers = [ ...
options = trainingOptions('adam', ...
'MaxEpochs',300, ...
'GradientThreshold',1, ...
'InitialLearnRate',1e-4, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropPeriod',125, ...
'LearnRateDropFactor',0.2, ...
'Verbose',0, ...
% X_train is a cell with length of 350, and each cell element is a [785,785,3] double matrix
% y_train is a cell with length of 350, and each cell element is a [785*785*3,1] double matrix
net = trainNetwork(X_train,y_train,layers,options);
for i=1:length(X_train)
[net,YPred] = predictAndUpdateState(net,X_train{i},'ExecutionEnvironment','cpu');
% forecast
Ypred = zeros(785,785,3,50);
[net,YPred] = predictAndUpdateState(net,YPred,'ExecutionEnvironment','cpu');
Ypred(:,:,:,1) = reshape(YPred,785,785,3);
for i=2:length(X_test)
[net,YPred] = predictAndUpdateState(net,Ypred(:,:,:,i-1),'ExecutionEnvironment','cpu');
Ypred(:,:,:,i) = reshape(YPred,785,785,3);
I compare the forecast result Ypred with my test data X_test, I got an undesirable compare result. I think the reason is I designed an inappropriate LSTM network. Any comments and suggestions are welcome! Thanks a lot!

Accepted Answer

Prateek Rai
Prateek Rai on 13 Sep 2021
Edited: Prateek Rai on 13 Sep 2021
To my understanding, you are trying to use the LSTM Network to forecast the image. A possible workaround could be :
(a) Use CNN Layers to extract features from image. The final result of Convolution must be a 1-D Vector.
(b) Use this 1-D vector to feed it to LSTM which will again output a new 1-D Vector.
(c) Output of LSTM can be feeded to a set of transposed convolutional layers to retrieve the output image.
You can refer to Convolutional Neural Network MathWorks documentation page to find more on CNN and transposedConv2dLayer MathWorks documentation page to find more on transposed Convolutional layers. You can also refer to Long Short-Term Memory Networks MathWorks documentation page to find more on LSTM.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!