Regression Learner and 1 layer Neural Network Model parameters

I used Regression Learner to generate a 1 fully-connected layer, narrow neural network with no activation function. The exported compact model works great when I run data through it. However, if I use the Layer Weights and Biases from the model to try and calculate the neural network myself, I am doing something wrong.

Here is what i've tried.

predictor1=rand(100,1);
predictor2=rand(100,1);
response=(2*predictor1 - 3*predictor2);% [predictor1 predictor2]*[2 -3]'

dataset=[predictor1 predictor2 response];

% Use Regression Learner App with dataset to train Narrow Neural Network
% activation function = none, 1 layer (size =1). 
% No surprise it has an RMSE approaching zero and R^2 =1. 
% Export & use compact model. 
yfit = trainedModel.predictFcn([predictor1 predictor2]);
disp(max (yfit - response)) % output 1.2234e-05  (small, as expected, model works)

%extract the parameters from the model
k1=trainedModel.RegressionNeuralNetwork.LayerWeights{1,1}; 
b1=trainedModel.RegressionNeuralNetwork.LayerBiases{1};
k2=trainedModel.RegressionNeuralNetwork.LayerWeights{1,2};
b2=trainedModel.RegressionNeuralNetwork.LayerBiases{2};

%calculate yfit outside of matlab's model
myfit=([predictor1 predictor2]*k1'+b1)*k2+b2; % WHAT AM I DOING WRONG???
disp(max (myfit - response)) % output 1.3108 (large, something is wrong with myfit)

What am I missing here?

Answer 1

[Andreas Apostolatos]

Hi,

The source of the discrepancy in this case comes from the fact, that you are performing the "manual" forward pass without any normalization of the data using the weights and biases of the shallow neural network that corresponds to the normalized data as obtained by the Regression Learner App in MATLAB, although you are training the shallow neural network model by standardizing the input data.

I tried the same procedure using your test data by means of the Regression Learner App in MATLAB. Please select the Narrow Neural Network and then open the model options in the Regression Learner App. Then, please set the option "Standardize data" to "No" and train your model, see the following screenshot,

Then, please export the model as compact into MATLAB Base Workspace and use the following code snippet to verify that the results of the 'predictFcn' from the exported 'trainedModel' variable are the same as when performing the inference (forward pass) "manually", namely,

load trainedModel.mat % load pretrained model from the attachment

predictor1=rand(100,1);
predictor2=rand(100,1);
response=(2*predictor1 - 3*predictor2);% [predictor1 predictor2]*[2 -3]'
% predictor = [predictor1 predictor2];
dataset=[predictor1 predictor2 response];
% Use Regression Learner App with dataset to train Narrow Neural Network
% activation function = none, 1 layer (size =1). 
% No surprise it has an RMSE approaching zero and R^2 =1. 
% Export & use compact model. 
yfit = trainedModel.predictFcn([predictor1 predictor2]); % trainedModel object does not use normalization of the data
disp(max (yfit - response)) % output 2.4192e-04

%extract the parameters from the model
k1=trainedModel.RegressionNeuralNetwork.LayerWeights{1,1}; 
b1=trainedModel.RegressionNeuralNetwork.LayerBiases{1};
k2=trainedModel.RegressionNeuralNetwork.LayerWeights{1,2};
b2=trainedModel.RegressionNeuralNetwork.LayerBiases{2};

%calculate yfit outside of matlab's model
myfit=([predictor1 predictor2]*k1'+b1')*k2'+b2; 
disp(max (myfit - response)) % output 2.4192e-04

Please download the attached mat-file trainedModel.mat, upload it in the "Run feature" from MATLAB Answers below, create a code snippet, copy and paste the code above, and finally run the code snippet to reproduce the findings listed above.

Regarding the normalization of the data, please visit the following documentation page,

https://www.mathworks.com/help/stats/choose-regression-model-options.html

I hope this helps you to proceed.

Kind regards,

Andreas

[SW]

Thank you for the quick response! I did not think to turn off the "standardize data" option. I'm up an running again.