Integrating a LSTM layer into a NARX network

Hi, is it possible to integrate an LSTM layer into this type of network?

obtaining a network like:

Input Layer - NARX - LSTM - Output Layer ?

thanks to anyone who can help me

I attach my current code where I would like to insert the LSTM layer:

_______________________________________________________________________________________

% Solve an Autoregression Problem with External Input with a NARX Neural Network

%

% This script assumes these variables are defined:

%

% NN-IN - input time series.

% NN-TARG - feedback time series.

clear; clc; format long;

IN = readmatrix('NN-IN.xlsx');

TARG = readmatrix('NN-TARG.csv');

X = tonndata(IN,false,false);

T = tonndata(TARG,false,false);

% Choose a Training Function

% 'trainscg' uses less memory. Suitable in low memory situations.

% 'traingdx' Gradient descent with momentum and adaptive learning rate backpropagation

trainFcn = 'trainscg'; % Scaled conjugate gradient backpropagation.

% Create a Nonlinear Autoregressive Network with External Input

inputDelays = 1:2;

feedbackDelays = 1:2;

hiddenLayerSize = [30,10];

net = narxnet(inputDelays,feedbackDelays,hiddenLayerSize,'open',trainFcn);

% Prepare the Data for Training and Simulation

[x,xi,ai,t] = preparets(net,X,{},T);

% Setup Division of Data for Training, Validation, Testing

net.divideFcn = 'divideblock';

net.divideMode = 'time';

net.divideParam.trainRatio = 70/100;

net.divideParam.valRatio = 15/100;

net.divideParam.testRatio = 15/100;

%Tolerance

net.trainParam.max_fail=3;

tic

% Train the Network

[net,tr] = train(net,x,t,xi,ai);

toc

% Test the Network

y = net(x,xi,ai);

e = gsubtract(t,y);

% net.performParam = 'normalized';

% net.performFcn = 'mse';

performance = perform(net,t,y);

% View the Network

view(net);

[sam l]

did you able to use LSTM and narx together

[GIROLAMO ODDO]

Hello Sam

No I would not know how to insert an lstm layer into a narx network, as the object properties (net=narxnet(...)) do not allow me to do this, it only allows me to add fully connected layers. If you could give me an example of how to do this you would be doing me a big favour. Thanks

[sam l]

I am looking to use NARX and LSTM , but yet to figure out it .

I was looking at CNN+RNN and thought if i can be done

https://www.mathworks.com/matlabcentral/fileexchange/91360-time-series-forecasting-using-hybrid-cnn-rnn

[GIROLAMO ODDO]

Thank you very much Sam

Answer 1

[Conor Daly]

Hi Girolamo,

You can combine NARX and LSTM architectures within dlnetwork. Note that a NARX network is essentially a 1D convolution over a concatenation of the input sequence x and the time steps t. Here's an example which you can run using the L-BFGS optimizer which was released with R2023a:

%% Get MagLev data.
[x,t] = maglev_dataset;
x = [x{:}];
t = [t{:}];

X1 = dlarray(x(:, 1:end-1), 'CTB');
X2 = dlarray(t(:, 1:end-1), 'CTB');
T = dlarray(t(:, 3:end), 'CTB');

%% Construct NARX-LSTM dlnetwork.
layers = [ sequenceInputLayer(1, Name="xin", MinLength=2) 
    concatenationLayer(1, 2, Name="cat")
    convolution1dLayer(2, 10)
    tanhLayer()
    lstmLayer(16)
    fullyConnectedLayer(1) ];
lg = layerGraph(layers);
lg = addLayers(lg, sequenceInputLayer(1, Name="tin", MinLength=2));
lg = connectLayers(lg, "tin", "cat/in2");
net = dlnetwork(lg);

analyzeNetwork(net, X1, X2)

%% Train using L-BFGS.
maxEpochs = 150;
solverState = [];
lossFcn = @(net)dlfeval(@modelLoss, net, X1, X2, T);

monitor = trainingProgressMonitor;
monitor.Metrics = "TrainingLoss";
monitor.XLabel = "Epoch";

for epoch = 1:maxEpochs
    [net, solverState] = lbfgsupdate(net, lossFcn, solverState);
    recordMetrics(monitor, epoch, TrainingLoss=solverState.Loss);
end

%% Open-loop inference.
Y = predict(net, X1, X2);

yopen = extractdata(Y(:));
figure;
plot(yopen, '.'), hold on, plot(t(3:end))

%% Model loss function.
function [loss, grad] = modelLoss(net, X1, X2, T)
Y = predict(net, X1, X2);
loss = l2loss(Y, T);
grad = dlgradient(loss, net.Learnables);
end