risk.validation.thresholdMetrics

Return threshold metrics for receiver operating characteristic curve

Since R2025a

Syntax

metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse)

metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse,Name=Value)

Description

metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse) returns a table of receiver operating characteristic (ROC) curve metrics, where Score contains numeric values that represent quantities such as rankings or predictions, probability of default (PD), or loss given default (LGD) estimates. For example, in credit scoring models, the values in Score can represent individual credit scores or other credit data. BinaryResponse specifies the target state of each value in Score.

metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse,Name=Value) specifies one or more optional name-value arguments to customize the table of ROC curve metrics. For example, metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse,SortDirection="ascending",Metrics=["TruePositiveRate" "FalsePositiveRate"]) sorts the values in Score and returns a table with the metrics TruePositiveRate and FalsePositiveRate.

Examples

Return Threshold Metrics for Credit Scores

Return threshold metrics for credit score data. In this example, you use the credit validation data set which includes a table, ScorecardValidationData, that contains credit scores and their corresponding default status information.

Load and display the credit validation data.

load CreditValidationData.mat
head(ScorecardValidationData)

    CreditScore      PD       Default
    ___________    _______    _______

      579.86       0.14182       0   
      563.65       0.17143       0   
      549.52       0.20106       0   
      546.25       0.20845       0   
      485.34       0.37991       0   
      482.07       0.39065       0   
      579.86       0.14182       1   
      451.73         0.494       0   

Extract the variables CreditScore and Default from the table ScorecardValidationData. Use Default as the BinaryResponse input argument.

Score = ScorecardValidationData.CreditScore;
BinaryResponse = ScorecardValidationData.Default;

Return a table of threshold metrics by using the thresholdMetrics function with the fully qualified namespace risk.validation. Since Score contains credit scores, where low values commonly correspond to higher risk individuals, you can set the sorting direction to "ascending" by using the SortDirection name-value argument. This setting ensures that TruePositiveRate represents the proportion of defaulters.

metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse,SortDirection="ascending")

metricsTable=107×8 table
    Threshold    TruePositiveRate    FalsePositiveRate    RateOfPositivePredictions    TruePositives    FalsePositives    TrueNegatives    FalseNegatives
    _________    ________________    _________________    _________________________    _____________    ______________    _____________    ______________

     408.99                 0                   0                        0                   0                 0               234              126      
     408.99          0.071429            0.012821                 0.033333                   9                 3               231              117      
     410.12          0.079365            0.017094                 0.038889                  10                 4               230              116      
     430.66          0.087302            0.017094                 0.041667                  11                 4               230              115      
     435.52          0.087302            0.025641                 0.047222                  11                 6               228              115      
     436.65           0.10317            0.029915                 0.055556                  13                 7               227              113      
     439.33           0.11905            0.029915                 0.061111                  15                 7               227              111      
     440.45           0.13492            0.029915                 0.066667                  17                 7               227              109      
     446.06           0.14286            0.047009                 0.080556                  18                11               223              108      
     447.19           0.15079            0.055556                 0.088889                  19                13               221              107      
     451.73            0.1746             0.07265                  0.10833                  22                17               217              104      
     453.88           0.21429            0.094017                  0.13611                  27                22               212               99      
        455           0.22222             0.10684                  0.14722                  28                25               209               98      
     458.54           0.23016             0.11966                  0.15833                  29                28               206               97      
     459.66           0.25397             0.12393                  0.16944                  32                29               205               94      
     465.86            0.2619             0.13248                  0.17778                  33                31               203               93      
      ⋮

The first row in metricsTable is the starting point of the data that you can use in making plots such as the ROC curve. The value of Threshold in this row corresponds to the minimum possible value in Score if your data is in ascending order, or the maximum possible value in Score if your data is in descending order.

Input Arguments

Score — Score values

numeric vector

Score values, specified as a numeric vector, containing values that indicate quantities such as a rankings or predictions, PD, or LGD estimates.

BinaryResponse — Binary response

numeric or logical vector, containing values of 1 (true) or 0 (false)

Binary response, specified as a numeric or logical vector, that contains values of 1 (true) or 0 (false). The binary response represents the target state for each value in Score. For example, you can use the binary response to represent a discretized LGD target, where ones indicate a high LGD value.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse,SortDirection="ascending")

SortDirection — Sorting direction of the unique values in Score

"descending" (default) | "ascending"

Sorting direction of the unique values in Score, specified as "descending" or "ascending". If Score contains credit scores, where low values commonly correspond to higher risk individuals, you can set the sorting direction to "ascending". This setting ensures that TruePositiveRate represents the proportion of defaulters. If Score contains PD values, where higher values correspond to higher risk, sorting the values in descending order is common practice.

Metrics — List of output metrics

["tpr" "fpr" "rpp" "tp" "fp" "tn" "fn"] (default) | string array

List of output metrics, specified as a string array. You can request any of the following metrics by specifying the names or abbreviations in this table:

Name	Abbreviation	Description
"TruePositives"	"tp"	Number of true positives (TP)
"False Negatives"	"fn"	Number of false negatives (FN)
"FalsePositives"	"fp"	Number of false positives (FP)
"TrueNegatives"	"tn"	Number of true negatives (TN)
"SumOfTrueAndFalsePositives"	"tp+fp"	Sum of TP and FP
"RateOfPositivePredictions"	"rpp"	Rate of positive predictions (RPP), (TP+FP)/(TP+FN+FP+TN)
"RateOfNegativePredictions"	"rnp"	Rate of negative predictions (RNP), (TN+FN)/(TP+FN+FP+TN)
"Accuracy"	"accu"	Accuracy, (TP+TN)/(TP+FN+FP+TN)
"TruePositiveRate"	"tpr"	True positive rate (TPR), sensitivity, recall, or probability of detection
"FalsePositiveRate"	"fpr"	False positive rate (FPR), false alarm rate, fallout, or 1-specificity
"FalseNegativeRate"	"fnr"	False negative rate (FNR), or miss rate, FN/(TP+FN)
"TrueNegativeRate"	"tnr"	True negative rate (TNR), or specificity, TN/(TN+FP)
"PositivePredictiveValue"	"ppv"	Positive predictive value (PPV), or precision, TP/(TP+FP)
"NegativePredictiveValue"	"npv"	Negative predictive value (NPV), TN/(TN+FN)
"ExpectedCost"	"ecost"	Expected cost (EC)
"F1score"	"f1score"	F1 score, 2*TP/(2*TP+FP+FN)
You can obtain all of the previous metrics by specifying "all". You cannot specify "all" with any other metric.

For example, metricsTable = risk.validation.thresholdMetrics(Score,BinaryResponse,Metrics=["fpr" "tpr"]) returns a table with the columns FalsePositiveRate and TruePositiveRate.

Output Arguments

metricsTable — ROC metrics table

table

ROC metrics, returned as a NumRows-by-NumCols table, where the first column, Thresholds, contains the score threshold values. Use the Metrics name-value argument to return the metrics you want.

If you do not specify the Metrics name-value argument, the function returns the following set of default metrics:

• TruePositiveRate — True positive rate values corresponding to the unique scores in the Thresholds column. For credit scoring models, this metric corresponds to the proportion of defaulters. This metric is also known as the sensitivity, recall, or probability of detection.

• FalsePositiveRate — False positive rate values corresponding to the unique scores in the Threshold column. For credit scoring models, this metric corresponds to the proportion of nondefaulters. This metric is also known as the false alarm rate, fallout, or 1 - specificity, where specificity is the true negative rate.

• RateOfPositivePredictions — Rate of positive predictions values corresponding to the unique scores in the Threshold column. This metric corresponds to the proportion of borrowers.

• TruePositives — Cumulative count of true positives corresponding to the threshold value in row i of the table. For credit scoring models, this metric corresponds to the number of defaulters in rows 1 to i-1.

• FalsePositives — Cumulative count of false positives corresponding to the threshold value in row i of the table. For credit scoring models, this metric corresponds to the number of nondefaulters in rows 1 to i-1.

• TrueNegatives — Total count of true negatives corresponding to the threshold value in row i of the table. For credit scoring models, this metric corresponds to the total number of nondefaulters contained in rows i+1 to NumRows.

• FalseNegatives — Total count of false negatives corresponding to the threshold value in row i of the table. For credit scoring models, this metric corresponds to the total number of defaulters contained in rows i+1 to NumRows.

Note

The first row in metricsTable is the starting point of the data that you can use in making plots such as the ROC curve. The value of Threshold in this row corresponds to the minimum possible value in Score if your data is in ascending order, or the maximum possible value in Score if your data is in descending order.

Version History

Introduced in R2025a