Neural network toolbox - initialize the weights and biases with INITZERO:

Dear all,

I use initzero to set all the weights and biases of a neural network equal to 0 before training. The training stops at the first or second iteration with all resulting weights which are unexpectedly 0. This is not correct. By the way, I used matlab 2014a and tried several train functions (trainlm, trainbr, trainscg...). Hope that you can give me a hint how to deal with this prolbem. Thanks for reading. My code is in the following:

[X,T] = maglev_dataset;

inputDelays = 1:3;  % input delay vector
outputDelays = 1:3;  % input delay vector
hiddenSizes = [ 3 2 2];   % network structure (number of neurons per hidden layer)
% training algorithm
trainFcn = 'trainlm';  % Levenberg-Marquardt
% trainFcn = 'trainbr';  % Levenberg-Marquardt

net = narxnet(inputDelays, outputDelays, hiddenSizes,'open',trainFcn);

%%define transfer function
net.layers{1}.transferFcn = 'tansig';
net.layers{2}.transferFcn = 'tansig';
net.layers{3}.transferFcn = 'tansig';
net.layers{4}.transferFcn = 'purelin';

%%prepare time-series for training network
[x,xi,ai,t] = preparets(net,X,{},T);

%%division scheme of data for validation, test...

net.divideFcn = 'divideblock';  % 
net.divideMode = 'value';  % Divide up every value
net.divideParam.trainRatio = 0.8;
net.divideParam.valRatio = 0.1;
net.divideParam.testRatio = 0.1;

%%training parameters
net.trainParam.showWindow = 1;

%%initialize all weights and biases before training

INITWEIGHTS = 1;

if INITWEIGHTS

net.initFcn = 'initlay';

for i=1:size(net.layers,1)
   net.layers{i}.initFcn = 'initwb'; 
end

   initialWeightsFunction = 'initzero';
  % initialWeightsFunction = 'midpoint';
  % initialWeightsFunction = 'rands';

   initialBiasesFunction = 'initzero';
  % initialBiasesFunction = 'rands';

for i=1:size(net.inputWeights,1)
   for j=1:size(net.inputWeights,2)
        if ~isempty(net.inputWeights{i,j})
            net.inputWeights{i,j}.initFcn = initialWeightsFunction;        
%             if initialWeightsFunction == 'midpoint'
%                net.inputWeights{i,j}.weightFcn = ''; 
%             end
        end
   end
end

for i=1:size(net.layerWeights,1)
   for j=1:size(net.layerWeights,2)
        if ~isempty(net.layerWeights{i,j})
            net.layerWeights{i,j}.initFcn = initialWeightsFunction;
        end
   end
end

for i=1:size(net.biases,1)

      if ~isempty(net.biases{i})
          net.biases{i}.initFcn = initialBiasesFunction;
      end

end

% initialize the weights and the biases of the network
net=init(net);

end

%%training network
[net, tr] = train(net,x,t,xi,ai);

[chu]

Greg Heath suggested me to control the random generator before initializing my network. This is an invaluable advice. However, I want to see what goes wrong when I initialize the weights with zero values.

[Greg Heath]

More later

Answer 1

[Greg Heath]

 clear all, clc
[ X,T ]         = maglev_dataset;
net             = narxnet;
[ Xs Xi Ai Ts ] = preparets(net,X,{},T);
ts     = cell2mat(Ts);
MSE00s = mean(var(ts',1)) % 2.0021 Reference MSE
rng('default')            % For repeatibility
net = configure(net,Xs,Ts);
Wb = getwb(net);           % size(Wb) =[ 61 1]
Nw = net.numWeightElements % 61
net = setwb(net,zeros(Nw,1));
[ net tr Ys Es Xf Af ] = train(net,Xs,Ts,Xi, Ai);
% Ys = net(Xs,Xi,Ai); Es = gsubtract(net,Ts,Ys);

 NMSEs = mse(Es)/MSE00s   % 1 ==> useless net!
stopcrit = tr.stop        %  Minimum gradient
numepochs = tr.num_epochs % 2

% To see most of the result details use the command

% tr = tr (No semicolon)

 IW11=net.IW{1,1}  % zeros(10,2)
IW12 = net.IW{1,2} % zeros(10,2)
b11 = net.b{1,1}   % zeros(10,1)
b21 = net.b{2,1}   %   -0.095645
LW21 = net.LW{2,1} % zeros(1,10)

Hope this helps.

Thank you for formally accepting my answer

Greg

[chu]

It's another perfect answer.

setwb is the command to be used instead of the "initzero".

By the way, I run the code you posted, and also see that the training stops at the second iteration. By commenting out the two lines (rng('default') and net=configure() ), I see that it's better.

Many thanks to Greg. Wish you all the best!!

cheers. Chu

[chu]

Hi again Greg,

I had a close look at your code. Now I think that using setwb one can set all the weights and biases from a given vector, but it is NOT THE INITIALIZATION OF THE WEIGHTS AND BIASES. when the network is trained, the initialization is done according to the scheme defined in net.layers{i}.initFcn , which is initnw by default.

Hope to hear some more advices and comments.

best regards. Chu