Neural network predicting impossible values

Hi.

I have created a NARX neural network that attempts to predict 96 values per day (daily values with a 15 minute-time resolution) -> The training target is a 1096x96 matrix (3 years) and the testing target is a 365x96 matrix (1 year).

I have normalized the data to [-1, 1] using mapminmax.apply(y,settings). settings was defined using the minimum and maximum possible raw values [0, 1500].

After training and optimizing so far, the network predicts normalized values below -1 and above 1, which it shouldn't.

Is there a way I can set boundaries for the output values? So far, I have had to interpolate the affected values, which is tedious and can mess up the results.

Thanks for any help!

- Marc

[Greg Heath]

Insufficient and confusing explanation.

Post your code to reduce the confusion and miscommunication.

Include the dimensions of each vector and matrix.

Are you using explicit or default variable normalization?

Include variable ranges minmax([input; target])

Using the result of the command

 who

should be helpful.

1. NARX is a time-series net that uses present and past inputs as well as past outputs to predict present and future outputs.

 net = narxnet(ID,FD,H);

what are ID, FD, H?

2. I assume you have input and output time-series with a 15 minute timestep.

a. What are you trying to predict every 15 minutes?
b. How far into the future
c. What are your I inputs
d. What are the dimensionalities of the I-dimensional input vector and the O-dimensional output vector?

      [ I N ] = size(input)
      [ O N ] = size(target) = size(output)

[Marc Jakobi]

Hi Greg. Thanks for getting back to me. I'll try to include as much as I can. I didn't include the code, because I am trying out various variations of the network to see what works best. The one I am currently working on looks like this:

Goal: Prediction of solar irradiance for 1 day (target) using inputs derived from pressure values of the previous day and the timestamp. The target dimensions are Nx4 (each time step N represents 1 hour with a 15 minute resolution - I want the network to predict 24 time steps, which I haven't got to yet) The input dimensions are also NxM (M varies with different variations, in my latest attempt I have reduced it to 4).

I have normalized the data in the following way before creating the input/target matrices:

[~,doyMM] = mapminmax([1 366],-1,1); %day of the year normalization settings
[~,E_MM] = mapminmax([0 1362],-1,1); %Irradiance normalized to solar constant

I normalize each input (Nx1 vector) separately and then combine them to the Nx4, since the normalization does not have to be reversed. I also normalize the target as an Nx1 matrix, before reshaping it to create the target matrix:

targetData = reshape(mapminmax.apply(E',E_MM),4,length(E)/4)'; %E = solar irradiance in W/m²

etc. So minmax([input, target]) is always -1 and 1.

After generating the forecast, I undo the normalization:

E_forecast = mapminmax.reverse(E_forecast,E_MM);

Regarding your question 1

ID = 1:24; % 24 past hours (I am still varying this, but currently, a one day delay seems to work best)
FD = 1:24; %24 past hours
H = 10;
net = narxnet(ID,FD,H,'open','trainlm');

I generate the forecast like this: (inputTestData is twice as long as targetTestData, since the inputs for the day to be forecasted are known.)

X = tonndata(inputTestData,false,false); %format inputs and targets for network
T = tonndata(targetTestData,false,false);
X1 = X(1:24); %input for feedback simulation
X2 = X(25:48); %input for forecast
[x,xi,ai,~] = preparets(net,X1,{},T);
[y1, xf, af] =  net(x,xi,ai); %get feedback from simulation
[netc, xi, ai] = closeloop(net,xf,af); %closed loop net for forecast %using feedbacks from simulation
[y2, xf, af] = netc(X2,xi,ai); %generate forecast
E_forecast= reshape(cell2mat(y2),4*23,1); %create vector from forecast
E_forecast = mapminmax.reverse(E_forecast',E_MM)'; %undo normalization

[Marc Jakobi]

The final goal is to create a system that generates a forecast for 24 hours ahead at midnight, then another one for 23 hours at 01:00 AM, then another for 22 hours at 02:00 AM, etc. so it updates the forecast for the rest of the day as measurements come in every hour.

[Greg Heath]

>Greg. Thanks for getting back to me. I'll try to include as much as I can. >I didn't include the code, because I am trying out various variations of >the network to see what works best.

OK. I can wait.

[Greg Heath]

It doesn't make sense to postulate the length of the forecast window when you haven't identified the significant delays of the output autocorrelation function and the input-output crosscorrelation functions.

In order to do this as well as identify and modify outliers, normalize by zscore and don't unnormalize until the very end of the program.

The default normalization/unnormalization for the net is mapminmax. If having 2 normalizations bothers you, you can remove it. However, it is easier to just keep it.

Contrary to what you have described, use a constant length forecast window of an hour so that

 [ 4 N-4 ] = size(target)

and, depending on the longest significant crosscorrelation delay

 [ I N-4 ] = size(input)

It may be helpful to first design a TIMEDELAYNET and a NARNET before designing the NARXNET.

It may also be helpful to practice on one or more of the MATLAB practice data sets

 help nndatasets
 doc nndatasets.

[Marc Jakobi]

The reasons I chose mapminmax over zscore for nomalization are because the inputs have extremely different variations (air pressure varies far less than the time stamp values, for example). Mapping each input to [-1, 1] should prevent important inputs from being "neglected". Also, since one weather station could have completely different irradiance values than the training/testing data set, I have to choose reference values for normalization (for example, the solar constant) so that I can easily undo the normalization. mapminmax seems to be the best solution so far, since reversing the normalization is less complicated and error-prone than other methods (please correct me if I am wrong). I only unnormalize the solar irradiance after the end of the program.

I am not quite sure what you mean by "use a constant length forecast window of an hour".

The results are generally acceptable. I have changed the number of delays to 96 (four days) and the adaption works quite well (it generates a forecast ahead for 24 time steps (1 day)) and then updates it every hour, forecasting ahead for an hour less every hour into the day until it "resets" the next day, forecasting 24 hours ahead again. I do not need a forecast for the next day at say, 12 PM on a specific date. All I need is a forecast for the remaining day. That's why I shorten the forecast length every hour into the day.

And for some reason, with 96 delays, there are still some values below -1, but far less than before (only at night, which is easy to correct) and not very far below anymore.

So my problem is not so drastic that I would have to rewrite the entire code. I was just wondering if there was something I could do to "let the network know" the minimum and maximum values it should output.

Answer 1

[Greg Heath]

If you must have normalized outputs in the range [-1,1],

Replace the default output transfer function of PURELIN with TANSIG.

Thank you for formally accepting my answer

Greg

[Greg Heath]

I recommend using zscore, minmax and plots BEFORE TRAINING to verify that both inputs and targets have reasonable summary statistics. If not, you can delete or modify outliers.

Then, you have the choice of using the already available zx and/or zt for training WITH OR WITHOUT the normalization in train.