MATLAB Answers

0

Neural Network (NARX) performance interpretation - how small is small?

Asked by Charles on 18 Aug 2017
Latest activity Answered by Greg Heath
on 19 Aug 2017
I have built a neural network to forecast currency pair prices (trend) a few steps ahead. How do I interpret the performance plot results? For instance, the error histogram, the NMSE (what is considered small for currency pairs? How do I assess if these metrics are optimal?

  0 Comments

Sign in to comment.

1 Answer

Answer by Greg Heath
on 19 Aug 2017

If you naively model a function with a straight line, the MSE is just the variance of the function and varies with the scale of the function.
In previous MATLAB posts I have used the notations
MSE00 = vart1 = mean(var(target'),1)
and use it as a standard for measuring MSE.
Using this NAIVE model as a comparative standard, the normalized MSE is defined by
NMSE = MSE/mean(var(target',1))
which is related to the statistical RSQUARE via
RSQUARE = 1 - NMSE
For reasonable models both NMSE and RSQUARE lie within the closed interval [ 0,1].
For most regression and classification models, I use a goal
NMSEgoal = 0.01 % Regression & Classification
However, in order to obtain that goal for CLOSED LOOP time series, I use the OPEN LOOP goals
NMSEgoal = 0.005 or 0.001
I don't know of anyone who complains when Rsquare >= 0.99 or equivalently, NMSE <= 0.01.
Hope this helps.
Thank you for formally accepting my answer
Greg

  0 Comments

Sign in to comment.