Is that possible to have the same training-MSEs for the same training data set with different weights at end of the training phase?

Matlab NN toolbox starts its weights and bias randomly. Therefore, I trained the network 200 times with random weights and bias to the same given training data set. I planed to get the minimum MSE out of 200 training MSEs, so that I can use the corresponding weights and bias in the testing phase. But in that case, I found it gives the same training-MSEs (most of the time) for the same training data set with different weight values at end of the training phase.

Is that possible? Can training stop at a local or global minima with different values for weights and bias?

really appreciate if anyone can clarify this issue.

Regards, Dara

Answer 1

[Greg Heath]

If you have H hidden nodes you can reorder them H! ways.

If you have an odd parity transfer function you can change the sign of its input weights and counter that by changing the sign of it's output weight.

Consequently, there are M = H! *2^H different nets that have identical input/output performance.

The default value of H = 10 yields M = 3.7159e+09.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Darshana Abeyrathna]

Hi Greg,

       Thank you for your answer :)

       Can I be more specific for my case? It has only 1 hidden layer (H = 1) as I use NNs for electricity forecasting. Then, each time interval has only one input-weight value to be trained. likewise, I use 192 inputs to the network and it has 48 outputs.

From your answer, I understood "odd parity transfer function" case. In that case, sign of the input-weights and output-weights could be change. But still thats not what I see with my results.

In " M = H! *2^H" case, what is H? is that number of hidden layers?. When we consider this as an optimization problem (reaching to the minimum training error), is that possible to have a local or global minima with different variable values?

Thank you very much!

Regards, Dara

[Greg Heath]

You misunderstand.

1. M is the number of rearrangements of the same weight values.

2. H is the number of nodes (neurons) in the single hidden layer. The MATLAB default is H = 10. How many input/target pairs do you have: N = ?

 [ I N ] = size(input) % [ 192 N ]
 [ O N ] = size(target)% [ 48 N ]

Assuming default trn/val/tst division ratios of 0.7/0.15/0.15, the number of training samples and equations are

 Ntrn   = N -2*round(0.15*N)) 
 Ntrneq = Ntrn*O

whereas the number of unknown weights is

 Nw = (I+1)*H+(H+1)*O = 1930+ 528 = 2458

Obviously, to have at least as many equations as unknowns you need AT LEAST

 N >~ 2458/48 ~ 51

3. You might want to reduce I=192 using PLSregress

Hope this helps.

Greg