Metrics Handlers
This example shows how to manage Modelscape™ test metrics and their associated threshold objects using MetricsHandler
objects.
The MetricsHandler
object produces reports that summarize the metrics and their status in the container relative to their thresholds.
For more information about test metrics and thresholds, see Credit Scorecard Validation Metrics and Fairness Metrics in Modelscape. To learn how to write your own metrics, see Test Metrics in Modelscape.
In this example, you set up metrics and thresholds for mock data of a credit scoring model. You create a MetricsHandler
object to visualize the metrics and summarize the results. You then set an overall status for the handler based on these metrics.
Set Up Test Metrics and Thresholds
Use the following random data as examples of training response data (defaultIndicators
) and model predictions (scores
).
rng("default");
scores = rand(1000,1);
defaultIndicators = double(scores + rand(1000,1) < 1);
Create these metrics:
Area under the receiver operating characteristic curve (AUROC)
Cumulative accuracy profile (CAP accuracy) ratio
Kolmogorov-Smirnov statistic
For the AUROC and CAP accuracy ratios, designate values greater than 0.8 as a pass, values less than 0.7 as a failure, and values between these as undecided, requiring further inspection. Set no thresholds for the Kolmogorov-Smirnov statistic.
import mrm.data.validation.TestThresholds import mrm.data.validation.pd.* auroc = AUROC(defaultIndicators,scores); aurocThresholds = TestThresholds([0.7 0.8],["Fail","Undecided","Pass"]); cap = CAPAccuracyRatio(defaultIndicators,scores); capThresholds = TestThresholds([0.6 0.7],["Fail","Undecided","Pass"]); ks = KSStatistic(defaultIndicators,scores);
Add Metrics to Metrics Handler Object
Add the metrics to a MetricsHandler
object and display the result.
import mrm.data.validation.MetricsHandler
mh = MetricsHandler;
append(mh,auroc,aurocThresholds);
append(mh,cap,capThresholds);
append(mh,ks);
disp(mh)
MetricsHandler with properties: AUROC: [1x1 mrm.data.validation.pd.AUROC] KS: [1x1 mrm.data.validation.pd.KSStatistic] CAP: [1x1 mrm.data.validation.pd.CAPAccuracyRatio]
The handler contains these three metrics you can access as properties of this handler object. Use these properties to access the constituent metrics diagnostics and visualizations.
visualize(mh.AUROC);
Interrogate Metrics Handlers
View the performance of the model relative to the given metrics by using the report method
.
The model performs well on AUROC, but the undecided status of the Accuracy Ratio suggests the model requires a closer look.
summaryTable = report(mh); disp(summaryTable)
Metric Value Status Diagnostic ____________________________ _______ ___________ ___________ Area under ROC curve 0.82905 Pass (0.8, Inf) Accuracy ratio 0.65809 Undecided (0.6, 0.7] Kolmogorov-Smirnov statistic 0.51462 <undefined> <undefined>
When the handler carries complex, non-scalar metrics, use Keys
and Metrics
arguments with report
. For more information, see Fairness Metrics in Modelscape.
Set Overall Status for the Handler
For a handler with many metrics, set an overall status for the handler by associating a status interpreter with the handler. In this example, you use a built-in Modelscape interpreter that is compatible with your threshold objects. The status descriptions of the individual metrics determine the overall status. In this case, the overall status is undecided, corresponding to the worst individual status.
mh.StatusInterpreter = @mrm.data.validation.overallStatus; summaryTable = report(mh); disp(summaryTable)
Metric Value Status Diagnostic ____________________________ _______ ___________ ___________ Area under ROC curve 0.82905 Pass (0.8, Inf) Accuracy ratio 0.65809 Undecided (0.6, 0.7] Kolmogorov-Smirnov statistic 0.51462 <undefined> <undefined> Overall NaN Undecided <undefined>
Implementing thresholding systems with other descriptive strings requires a custom status interpreter. See the instructions before the StatusInterpreter declaration in the MetricsHandler implementation.
edit mrm.data.validation.MetricsHandler
Alternatively, modify the interpreter for your needs.
edit mrm.data.validation.overallStatus
You can also set the StatusInterpreter
property of the handler when you create the object, using this command.
mh2 = MetricsHandler(StatusInterpreter=@mrm.data.validation.overallStatus)