No information rate test

Hi, I'm here to ask you if there exists something similar to the no information rate test in matlab, I want to explain myself better: during classification analysis I met the need to statistically compute the p-value of a function that allows me to test the hypothesis that the accuracy (true predicted label / true label) is actually better than no information rate (is the proportion of the largest class within the dataset)

Answer 1

[Aneela]

Hi Alberto Azzari,

In MATLAB, there isn't a built-in function that directly computes the p-value to test whether the accuracy of a classifier is significantly better than the no information rate (NIR).

However, you can refer to the workflow below for Permutation Testing:

• Compute the No Information Rate (NIR) by finding the proportion of the largest class in your dataset.
• Compute the “observed accuracy” of the classifier on the original dataset.
• Randomly permute the class labels of your dataset many times, each time calculating the accuracy of your classifier with the permuted labels.
• This will give you a distribution of accuracies under Null Hypothesis.
• The p-value is the proportion of accuracies from the permutation test that are equal to or greater than the observed accuracy.
• A low p-value suggests that the observed accuracy is significantly better than NIR.

To prove that a model is significant, the accuracy should be higher than the NIR.