addMetrics

Compute additional classification performance metrics

Since R2022a

Syntax

UpdatedROCObj = addMetrics(rocObj,metrics)

Description

rocmetrics computes the false positive rates (FPR), true positive rates (TPR), and additional metrics specified by the AdditionalMetrics name-value argument. After creating a rocmetrics object, you can compute additional classification performance metrics by using the addMetrics function.

UpdatedROCObj = addMetrics(rocObj,metrics) computes additional classification performance metrics specified in metrics using the classification model information stored in the rocmetrics object rocObj.

UpdatedROCObj contains all the information in rocObj plus additional performance metrics computed by addMetrics. The function attaches the additional computed metrics (metrics) as new variables in the table of the Metrics property.

If you compute confidence intervals when you create rocObj, the addMetrics function computes the confidence intervals for the additional metrics. The new variables in the Metrics property contain a three-column matrix in which the first column corresponds to the metric values, and the second and third columns correspond to the lower and upper bounds, respectively.

Examples

Compute Additional Metrics

Compute the performance metrics (FPR, TPR, and expected cost) for a multiclass classification problem when you create a rocmetrics object. Compute additional metrics, the positive predictive value (PPV) and the negative predictive value (NPV), and add them to the object.

Load the fisheriris data set. The matrix meas contains flower measurements for 150 different flowers. The vector species lists the species for each flower. species contains three distinct flower names.

load fisheriris

Train a classification tree that classifies observations into one of the three labels. Cross-validate the model using 10-fold cross-validation.

rng("default") % For reproducibility
Mdl = fitctree(meas,species,Crossval="on");

Compute the classification scores for validation-fold observations.

[~,Scores] = kfoldPredict(Mdl);
size(Scores)

ans = 1×2

   150     3

Scores is a matrix of size 150-by-3. The column order of Scores follows the class order in Mdl. Display the class order stored in Mdl.ClassNames.

Mdl.ClassNames

ans = 3×1 cell
    {'setosa'    }
    {'versicolor'}
    {'virginica' }

Create a rocmetrics object by using the true labels in species and the classification scores in Scores. Specify the column order of Scores using Mdl.ClassNames. By default, rocmetrics computes the FPR and TPR. Specify AdditionalMetrics="ExpectedCost" to compute the expected cost as well.

rocObj = rocmetrics(species,Scores,Mdl.ClassNames, ...
    AdditionalMetrics="ExpectedCost");

The table in the Metrics property of rocObj contains performance metric values for all three classes, vertically concatenated according to the class order. Find and display the rows for the second class in the table.

idx = strcmp(rocObj.Metrics.ClassName,Mdl.ClassNames(2));
rocObj.Metrics(idx,:)

ans=13×5 table
      ClassName       Threshold    FalsePositiveRate    TruePositiveRate    ExpectedCost
    ______________    _________    _________________    ________________    ____________

    {'versicolor'}           1              0                    0            0.074074  
    {'versicolor'}           1           0.01                  0.7            0.023704  
    {'versicolor'}     0.95455           0.02                  0.8            0.017778  
    {'versicolor'}     0.91304           0.03                  0.9            0.011852  
    {'versicolor'}        -0.2           0.04                  0.9            0.013333  
    {'versicolor'}    -0.33333           0.06                  0.9            0.016296  
    {'versicolor'}        -0.6           0.08                  0.9            0.019259  
    {'versicolor'}    -0.86957           0.12                 0.92            0.023704  
    {'versicolor'}    -0.91111           0.16                 0.96            0.026667  
    {'versicolor'}    -0.95122           0.31                 0.96            0.048889  
    {'versicolor'}    -0.95238           0.38                 0.98            0.057778  
    {'versicolor'}    -0.95349           0.44                 0.98            0.066667  
    {'versicolor'}          -1              1                    1             0.14815  

The table in Metrics contains the variables for the class names, threshold, false positive rate, true positive rate, and expected cost (the additional metric).

After creating a rocmetrics object, you can compute additional metrics using the classification model information stored in the object. Compute the PPV and NPV by using the addMetrics function. To overwrite the input argument rocObj, assign the output of addMetrics to the input.

rocObj = addMetrics(rocObj,["PositivePredictiveValue","NegativePredictiveValue"]);

Display the Metrics property.

rocObj.Metrics(idx,:)

ans=13×7 table
      ClassName       Threshold    FalsePositiveRate    TruePositiveRate    ExpectedCost    PositivePredictiveValue    NegativePredictiveValue
    ______________    _________    _________________    ________________    ____________    _______________________    _______________________

    {'versicolor'}           1              0                    0            0.074074                  NaN                    0.66667        
    {'versicolor'}           1           0.01                  0.7            0.023704              0.97222                    0.86842        
    {'versicolor'}     0.95455           0.02                  0.8            0.017778              0.95238                    0.90741        
    {'versicolor'}     0.91304           0.03                  0.9            0.011852               0.9375                    0.95098        
    {'versicolor'}        -0.2           0.04                  0.9            0.013333              0.91837                     0.9505        
    {'versicolor'}    -0.33333           0.06                  0.9            0.016296              0.88235                    0.94949        
    {'versicolor'}        -0.6           0.08                  0.9            0.019259              0.84906                    0.94845        
    {'versicolor'}    -0.86957           0.12                 0.92            0.023704               0.7931                    0.95652        
    {'versicolor'}    -0.91111           0.16                 0.96            0.026667                 0.75                    0.97674        
    {'versicolor'}    -0.95122           0.31                 0.96            0.048889              0.60759                    0.97183        
    {'versicolor'}    -0.95238           0.38                 0.98            0.057778              0.56322                    0.98413        
    {'versicolor'}    -0.95349           0.44                 0.98            0.066667              0.52688                    0.98246        
    {'versicolor'}          -1              1                    1             0.14815              0.33333                        NaN        

The table in Metrics now includes the PositivePredictiveValue and NegativePredictiveValue variables in the last two columns, in the order you specified. Note that the positive predictive value (PPV = TP/(TP+FP)) is NaN for the reject-all threshold (largest threshold), and the negative predictive value (NPV = TN/(TN+FN)) is NaN for the accept-all threshold (lowest threshold). TP, FP, TN, and FN represent the number of true positives, false positives, true negatives, and false negatives, respectively.

Input Arguments

rocObj — Object evaluating classification performance

rocmetrics object

Object evaluating classification performance, specified as a rocmetrics object.

metrics — Additional model performance metrics

character vector | string array | function handle | cell array

Additional model performance metrics to compute, specified as a character vector or string scalar of the built-in metric name, string array of names, function handle (@metricName), or cell array of names or function handles. A rocmetrics object always computes the false positive rates (FPR) and the true positive rates (TPR) to obtain a ROC curve. Therefore, you do not have to specify to compute FPR and TPR.

• Built-in metrics — Specify one of the following built-in metric names by using a character vector or string scalar. You can specify more than one by using a string array.

  Name	Description
  "TruePositives" or "tp"	Number of true positives (TP)
  "FalseNegatives" or "fn"	Number of false negatives (FN)
  "FalsePositives" or "fp"	Number of false positives (FP)
  "TrueNegatives" or "tn"	Number of true negatives (TN)
  "SumOfTrueAndFalsePositives" or "tp+fp"	Sum of TP and FP
  "RateOfPositivePredictions" or "rpp"	Rate of positive predictions (RPP), (TP+FP)/(TP+FN+FP+TN)
  "RateOfNegativePredictions" or "rnp"	Rate of negative predictions (RNP), (TN+FN)/(TP+FN+FP+TN)
  "Accuracy" or "accu"	Accuracy, (TP+TN)/(TP+FN+FP+TN)
  "FalseNegativeRate", "fnr", or "miss"	False negative rate (FNR), or miss rate, FN/(TP+FN)
  "TrueNegativeRate", "tnr", or "spec"	True negative rate (TNR), or specificity, TN/(TN+FP)
  "PositivePredictiveValue", "ppv", "prec", or "precision"	Positive predictive value (PPV), or precision, TP/(TP+FP)
  "NegativePredictiveValue" or "npv"	Negative predictive value (NPV), TN/(TN+FN)
  "ExpectedCost" or "ecost"	Expected cost, (TP*cost(P|P)+FN*cost(N|P)+FP*cost(P|N)+TN*cost(N|N))/(TP+FN+FP+TN), where cost is a 2-by-2 misclassification cost matrix containing [0,cost(N|P);cost(P|N),0]. cost(N|P) is the cost of misclassifying a positive class (P) as a negative class (N), and cost(P|N) is the cost of misclassifying a negative class as a positive class. The software converts the K-by-K matrix specified by the Cost name-value argument of rocmetrics to a 2-by-2 matrix for each one-versus-all binary problem. For details, see Misclassification Cost Matrix.
  "f1score"	F1 score, 2*TP/(2*TP+FP+FN)
  You can obtain all of the previous metrics by specifying "all". You cannot specify "all" in conjunction with any other metric.

  The software computes the scale vector using the prior class probabilities (Prior) and the number of classes in Labels, and then scales the performance metrics according to this scale vector. For details, see Performance Metrics.

• Custom metric — Specify a custom metric by using a function handle. A custom function that returns a performance metric must have this form:

  metric = customMetric(C,scale,cost)

  • The output argument metric is a scalar value.

  • A custom metric is a function of the confusion matrix (C), scale vector (scale), and cost matrix (cost). The software finds these input values for each one-versus-all binary problem. For details, see Performance Metrics.

    • C is a 2-by-2 confusion matrix consisting of [TP,FN;FP,TN].

    • scale is a 2-by-1 scale vector.

    • cost is a 2-by-2 misclassification cost matrix.

  The software does not support cross-validation for a custom metric. Instead, you can specify to use bootstrap when you create a rocmetrics object.

Note that the positive predictive value (PPV) is NaN for the reject-all threshold for which TP = FP = 0, and the negative predictive value (NPV) is NaN for the accept-all threshold for which TN = FN = 0. For more details, see Thresholds, Fixed Metric, and Fixed Metric Values.

Example: ["Accuracy","PositivePredictiveValue"]

Example: {"Accuracy",@m1,@m2} specifies the accuracy metric and the custom metrics m1 and m2 as additional metrics. addMetrics stores the custom metric values as variables named CustomMetric1 and CustomMetric2 in the Metrics property.

Data Types: char | string | cell | function_handle

Output Arguments

UpdatedROCObj — Object evaluating classification performance

rocmetrics object

Object evaluating classification performance, returned as a rocmetrics object.

To overwrite the input argument rocObj, assign the output of addMetrics to rocObj:

rocObj = addMetrics(rocObj,metrics);

Version History

Introduced in R2022a

R2024b: Additional metrics available

addmetrics has new metrics:

• "f1score", which computes the F1 score.

• "precision", which is the same as "ppv" and "prec".

• "all", which computes all supported metrics. You cannot use "all" in combination with any other metric.