Perform Feature Selection and Model Training for Transmission System Fault Detection
R2026bThis example shows how to train two classifiers: one for detecting sensor drift and one for detecting shaft wear in an automotive transmission system. In order to reduce the memory footprint of the two models, use feature selection to retain only the most important features, and tune hyperparameters with model size constraints. Verify that the final models satisfy the requirements outlined in Define Requirements for Transmission System Fault Detection.
Load Partitioned Tabular Data
Load the data variables in PartitionedFaultData.mat. The data set consists of features extracted from signal data, reformatted into tabular data, and partitioned into training and test sets. For more information, see Extract Features and Partition Data for Transmission System Fault Detection.
You do not need to load the data if you followed the previous step and have the faultDataTrain, faultDataTest, SensorDriftTrain, SensorDriftTest, ShaftWearTrain, and ShaftWearTest variables in your workspace already.
if ~exist("faultDataTrain","var") load("PartitionedFaultData.mat","faultDataTrain"); end if ~exist("faultDataTest","var") load("PartitionedFaultData.mat","faultDataTest"); end if ~exist("SensorDriftTrain","var") load("PartitionedFaultData.mat","SensorDriftTrain"); end if ~exist("SensorDriftTest","var") load("PartitionedFaultData.mat","SensorDriftTest"); end if ~exist("ShaftWearTrain","var") load("PartitionedFaultData.mat","ShaftWearTrain"); end if ~exist("ShaftWearTest","var") load("PartitionedFaultData.mat","ShaftWearTest"); end
The faultDataTrain and faultDataTest data sets contain the features that can be used as predictors when fitting the fault detection classifiers. Remove the ScenarioID feature from the data sets because it includes response variable information.
excludeFeatures = "ScenarioID";
excludeFeatures = intersect(excludeFeatures,faultDataTrain.Properties.VariableNames);
faultDataTrain(:,excludeFeatures) = [];
faultDataTest(:,excludeFeatures) = [];Display the number of observations and features in the predictor data sets.
numTrainingObservations = height(faultDataTrain)
numTrainingObservations = 1287
numTestObservations = height(faultDataTest)
numTestObservations = 551
numFeatures = width(faultDataTrain)
numFeatures = 19
Detect Sensor Drift Fault
To detect sensor drift faults, train a decision tree classifier. Decision trees are adept at modeling the threshold-based separations produced by sensor drift fault signatures. Additionally, decision trees are highly interpretable and have relatively small deployment footprints.
Train Decision Tree with Default Hyperparameters
First, train a decision tree classifier using the default hyperparameter values. Specify faultDataTrain as the predictor data and SensorDriftTrain as the response variable.
defaultSDClassifier = fitctree(faultDataTrain,SensorDriftTrain);
Compute the 5-fold cross-validation accuracy, and display the confusion matrix.
rng(0,"twister") % For reproducibility cvDefaultSDClassifier = crossval(defaultSDClassifier,KFold=5); sprintf("Accuracy: %f",1 - kfoldLoss(cvDefaultSDClassifier))
ans = "Accuracy: 0.998446"
cvPred = kfoldPredict(cvDefaultSDClassifier); confusionchart(SensorDriftTrain,cvPred)
The classifier cvDefaultSDClassifier returns near-perfect predictions on the validation folds, with few false positives.
Train Decision Tree Using Most Important Features
To reduce unnecessary feature extraction computations during deployment, limit the features used to train the classifier to those with high predictive power.
Determine the importance of the features used to train defaultSDClassifier by using permutationImportance. The function computes the importance of each feature in the model by permuting the values in the feature and comparing the model resubstitution loss with the original feature to the loss with the permuted feature. A large increase in the model loss with the permuted feature indicates that the feature is important.
rng(0,"twister") % For reproducibility SDFeatureImportance = permutationImportance(defaultSDClassifier); SDFeatureImportanceSorted = sortrows(SDFeatureImportance, ... ImportanceMean="descend")
SDFeatureImportanceSorted = 19×3 table
Predictor ImportanceMean ImportanceStandardDeviation
__________________ ______________ ___________________________
"SigPeak" 0.24709 0.0028372
"SigCrestFactor" 0.24468 0.0058771
"SigRMS" 0.018881 0.00082312
"SigMean" 0.0079254 0.0028794
"SigMedian" 0 0
"SigVar" 0 0
"SigPeak2Peak" 0 0
"SigSkewness" 0 0
"SigKurtosis" 0 0
"SigMAD" 0 0
"SigRangeCumSum" 0 0
"SigApproxEntropy" 0 0
"SigLyapExponent" 0 0
"PeakFreq" 0 0
"PeakSpecKurtosis" 0 0
"RPMMedian" 0 0
⋮
Select the top two most important features, and then retrain the decision tree classifier using only those features.
SDFeatureSubset = SDFeatureImportanceSorted.Predictor(1:2);
smallSDClassifier = fitctree(faultDataTrain(:,SDFeatureSubset), ...
SensorDriftTrain);Compute the 5-fold cross-validation accuracy.
rng(0,"twister") % For reproducibility cvSmallSDClassifier = crossval(smallSDClassifier,KFold=5); sprintf("Accuracy: %f",1 - kfoldLoss(cvSmallSDClassifier))
ans = "Accuracy: 0.991453"
The cross-validation accuracy remains high, even after training the model on only the two most important features.
Optimize and Train Decision Tree with Size Constraints
You can perform Bayesian optimization to find hyperparameter values that maximize the accuracy of your model while respecting model size constraints. This process is particularly useful for embedded deployment where RAM is limited. In this example, ensure the final model can be compiled to C/C++ code within a 10KB memory budget.
As before, use only the two most important features to train the model. Specify to optimize all hyperparameters, and in the optimization options, set AcquisitionFunctionName to "expected-improvement-plus" for reproducible results.
hpoOptions = hyperparameterOptimizationOptions(ConstraintTarget="coder", ... ConstraintType="size",ConstraintBounds=[0 10000],ShowPlots=false, ... AcquisitionFunctionName="expected-improvement-plus"); rng(0,"twister"); optimizedSDClassifier = fitctree(faultDataTrain(:,SDFeatureSubset), ... SensorDriftTrain,OptimizeHyperparameters="all", ... hyperparameterOptimizationOptions=hpoOptions);
|==================================================================================================================================================|
| |
|Objective : "kfoldLoss" |
|Constraint : "LearnerForCoderSize (bytes)" |
|Constraint Bounds : [0 10000] |
| |
|==================================================================================================================================================|
| Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Constraint1 | MinLeafSize | MaxNumSplits | SplitCriteri-| NumVariables-|
| | result | | runtime | (observed) | (estim.) | violation | | | on | ToSample |
|==================================================================================================================================================|
| 1 | Best | 0.006216 | 0.56706 | 0.006216 | 0.006216 | -8.84e+03 | 1 | 20 | deviance | 2 |
| 2 | Accept | 0.031857 | 0.20254 | 0.006216 | 0.008056 | -9.42e+03 | 343 | 13 | gdi | 2 |
| 3 | Accept | 0.031857 | 0.15797 | 0.006216 | 0.0062177 | -9.42e+03 | 47 | 2 | gdi | 2 |
| 4 | Accept | 0.031857 | 0.042647 | 0.006216 | 0.0062186 | -9.42e+03 | 338 | 268 | deviance | 2 |
| 5 | Accept | 0.010878 | 0.11402 | 0.006216 | 0.0062222 | -8.84e+03 | 3 | 253 | deviance | 2 |
| 6 | Accept | 0.006216 | 0.034239 | 0.006216 | 0.0062193 | -8.84e+03 | 1 | 256 | deviance | 2 |
| 7 | Accept | 0.006216 | 0.030298 | 0.006216 | 0.0062182 | -8.84e+03 | 1 | 31 | deviance | 2 |
| 8 | Accept | 0.031857 | 0.027882 | 0.006216 | 0.005868 | -9.42e+03 | 1 | 1 | deviance | 2 |
| 9 | Accept | 0.006216 | 0.029721 | 0.006216 | 0.006196 | -8.84e+03 | 1 | 91 | deviance | 2 |
| 10 | Accept | 0.006216 | 0.030751 | 0.006216 | 0.0060559 | -8.84e+03 | 1 | 1268 | deviance | 2 |
| 11 | Accept | 0.008547 | 0.029764 | 0.006216 | 0.0060902 | -8.26e+03 | 1 | 1243 | gdi | 2 |
| 12 | Accept | 0.027195 | 0.031379 | 0.006216 | 0.0060656 | -9.42e+03 | 29 | 1239 | gdi | 2 |
| 13 | Accept | 0.008547 | 0.02929 | 0.006216 | 0.0060268 | -8.26e+03 | 1 | 129 | gdi | 2 |
| 14 | Accept | 0.027972 | 0.027859 | 0.006216 | 0.0060625 | -9.19e+03 | 1 | 5 | gdi | 2 |
| 15 | Accept | 0.008547 | 0.029636 | 0.006216 | 0.0060381 | -8.26e+03 | 1 | 387 | gdi | 2 |
| 16 | Accept | 0.006216 | 0.029278 | 0.006216 | 0.0061714 | -8.84e+03 | 1 | 631 | deviance | 2 |
| 17 | Accept | 0.010878 | 0.031407 | 0.006216 | 0.0061259 | -8.84e+03 | 3 | 20 | deviance | 2 |
| 18 | Accept | 0.006216 | 0.034133 | 0.006216 | 0.0061927 | -8.84e+03 | 1 | 32 | deviance | 2 |
| 19 | Accept | 0.006216 | 0.02851 | 0.006216 | 0.0061899 | -8.84e+03 | 1 | 154 | deviance | 2 |
| 20 | Accept | 0.446 | 0.039496 | 0.006216 | 0.0060544 | -9.42e+03 | 635 | 1 | deviance | 2 |
|==================================================================================================================================================|
| Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Constraint1 | MinLeafSize | MaxNumSplits | SplitCriteri-| NumVariables-|
| | result | | runtime | (observed) | (estim.) | violation | | | on | ToSample |
|==================================================================================================================================================|
| 21 | Accept | 0.031857 | 0.028513 | 0.006216 | 0.0060882 | -9.42e+03 | 85 | 1285 | deviance | 2 |
| 22 | Accept | 0.446 | 0.026512 | 0.006216 | 0.0056108 | -9.42e+03 | 626 | 1279 | deviance | 2 |
| 23 | Accept | 0.031857 | 0.030755 | 0.006216 | 0.0056201 | -9.42e+03 | 37 | 150 | gdi | 2 |
| 24 | Accept | 0.031857 | 0.028463 | 0.006216 | 0.0056315 | -9.42e+03 | 243 | 451 | deviance | 2 |
| 25 | Accept | 0.009324 | 0.034464 | 0.006216 | 0.0056413 | -8.84e+03 | 2 | 1279 | deviance | 2 |
| 26 | Accept | 0.012432 | 0.029629 | 0.006216 | 0.0056515 | -8.95e+03 | 5 | 19 | deviance | 2 |
| 27 | Accept | 0.017094 | 0.039298 | 0.006216 | 0.0056604 | -9.07e+03 | 8 | 1233 | deviance | 2 |
| 28 | Accept | 0.031857 | 0.02854 | 0.006216 | 0.0056687 | -9.42e+03 | 13 | 1 | deviance | 2 |
| 29 | Accept | 0.031857 | 0.031017 | 0.006216 | 0.005678 | -9.07e+03 | 46 | 120 | deviance | 2 |
| 30 | Accept | 0.019425 | 0.033806 | 0.006216 | 0.0056853 | -9.07e+03 | 24 | 797 | deviance | 2 |
__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 6.6199 seconds
Total objective function evaluation time: 1.8589
Best observed feasible point:
MinLeafSize MaxNumSplits SplitCriterion NumVariablesToSample
___________ ____________ ______________ ____________________
1 20 deviance 2
Observed objective function value = 0.006216
Estimated objective function value = 0.0072122
Function evaluation time = 0.56706
Observed constraint violations =[ -8838.500000 ]
Best estimated feasible point (according to models):
MinLeafSize MaxNumSplits SplitCriterion NumVariablesToSample
___________ ____________ ______________ ____________________
1 91 deviance 2
Estimated objective function value = 0.0056853
Estimated function evaluation time = 0.040334
Estimated constraint violations =[ -8838.925420 ]
optimizedSDClassifier = optimizedSDClassifier{1};Compute the 5-fold cross-validation accuracy.
rng(0,"twister") % For reproducibility cvOptimizedSDClassifier = crossval(optimizedSDClassifier,KFold=5); sprintf("Accuracy: %f",1 - kfoldLoss(cvOptimizedSDClassifier))
ans = "Accuracy: 0.993784"
The optimized model retains a high cross-validation accuracy.
Compare Classifier Performance and Select Best Model
Compare the performance of the three sensor drift classifiers on the test data (faultDataTest and SensorDriftTest). Use the helper function helperComputeMetrics to compute the accuracy, true positive rate, false positive rate, false positives, false negatives, and code generation model size for each model.
SDClassifiers = {defaultSDClassifier,smallSDClassifier, ...
optimizedSDClassifier};
ModelNames = ["Default model","Model with selected features", ...
"Optimized model with selected features"];
SDClassifiersMetricsTable = helperComputeMetrics(SDClassifiers, ...
faultDataTest,SensorDriftTest,ModelNames)SDClassifiersMetricsTable = 3×6 table
Accuracy True Positive Rate False Positive Rate False Positives False Negatives Model Size (KBs)
________ __________________ ___________________ _______________ _______________ ________________
Default model 0.99636 0.99184 0 0 2 3.549
Model with selected features 0.99819 1 0.003268 1 0 1.742
Optimized model with selected features 0.99819 1 0.003268 1 0 1.162
The metrics table indicates that all three decision trees perform well on the test data. However, the models vary in the estimated amount of system memory they use in generated code.
Select optimizedSDClassifier as the sensor drift detector in the transmission system due to its minimal deployment footprint.
SDClassifier = optimizedSDClassifier;
Detect Shaft Wear Fault
To detect shaft wear faults, train an ensemble of trees. Unlike sensor drift, shaft wear is not well separated by simple threshold splits on individual features. For this reason, a decision tree is likely not sufficient for detecting shaft wear faults, and an ensemble of trees is a better classifier choice.
Train Ensemble with Default Hyperparameters
Train a tree-based ensemble using the default hyperparameter values. By default, fitcensemble uses the LogitBoost aggregate method with 100 tree learners.
rng(0,"twister") % For reproducibility defaultSWClassifier = fitcensemble(faultDataTrain,ShaftWearTrain);
Compute the 5-fold cross-validation accuracy.
cvDefaultSWClassifier = crossval(defaultSWClassifier,KFold=5);
sprintf("Accuracy: %f",1 - kfoldLoss(cvDefaultSWClassifier))ans = "Accuracy: 0.909868"
The classifier cvDefaultSWClassifier produces a cross-validation accuracy just over 90%.
Train Ensemble Using Most Important Features
Determine the importance of the features used to train defaultSWClassifier by using permutationImportance.
rng(0,"twister") % For reproducibility SWFeatureImportance = permutationImportance(defaultSWClassifier); SWFeatureImportanceSorted = sortrows(SWFeatureImportance, ... ImportanceMean="descend")
SWFeatureImportanceSorted = 19×3 table
Predictor ImportanceMean ImportanceStandardDeviation
__________________ ______________ ___________________________
"RPMStd" 0.12455 0.0061296
"RPMIQR" 0.11927 0.0060547
"RPMMedian" 0.070396 0.0036481
"SigLyapExponent" 0.029992 0.0020459
"SigPeak2Peak" 0.0034188 0.00083525
"PeakSpecKurtosis" 0.0022533 0.0013433
"SigSkewness" 0.0004662 0.00054328
"SigMean" 0.0003885 0.00040952
"SigApproxEntropy" 0.0001554 0.00032761
"SigVar" 0.0001554 0.00032761
"PeakFreq" 0.0001554 0.00032761
"SigKurtosis" 7.77e-05 0.00024571
"SigMedian" 0 0
"SigRMS" 0 0
"SigPeak" 0 0
"SigCrestFactor" 0 0
⋮
Select the top two most important features, and then retrain the ensemble using only those features.
SWFeatureSubset = SWFeatureImportanceSorted.Predictor(1:2); rng(0,"twister") % For reproducibility smallSWClassifier = fitcensemble(faultDataTrain(:,SWFeatureSubset), ... ShaftWearTrain);
Compute the 5-fold cross-validation accuracy.
cvSmallSWClassifier = crossval(smallSWClassifier,KFold=5);
sprintf("Accuracy: %f",1 - kfoldLoss(cvSmallSWClassifier))ans = "Accuracy: 0.929183"
After training the model on only the two most important features, the cross-validation accuracy has improved to be above the 92% threshold.
Optimize and Train Ensemble with Size Constraints
Perform Bayesian optimization to find hyperparameter values that maximize model accuracy while ensuring the final model can be compiled to C/C++ code within a 500KB memory budget.
Use only the two most important features to train the model, and specify to optimize all hyperparameters. In the optimization options, set AcquisitionFunctionName to "expected-improvement-plus" for reproducible results and increase the maximum number of iterations to 50 for better results.
hpoOptions = hyperparameterOptimizationOptions(ConstraintTarget="coder", ... ConstraintType="size",ConstraintBounds=[0 500000],ShowPlots=false, ... AcquisitionFunctionName="expected-improvement-plus", ... MaxObjectiveEvaluations=50); rng(0,"twister"); optimizedSWClassifier = fitcensemble(faultDataTrain(:,SWFeatureSubset), ... ShaftWearTrain,OptimizeHyperparameters="all", ... hyperparameterOptimizationOptions=hpoOptions);
|===============================================================================================================================================================================================|
| |
|Objective : "kfoldLoss" |
|Constraint : "LearnerForCoderSize (bytes)" |
|Constraint Bounds : [0 500000] |
| |
|===============================================================================================================================================================================================|
| Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Constraint1 | Method | NumLearningC-| LearnRate | MinLeafSize | MaxNumSplits | SplitCriteri-| NumVariables-|
| | result | | runtime | (observed) | (estim.) | violation | | ycles | | | | on | ToSample |
|===============================================================================================================================================================================================|
| 1 | Best | 0.34241 | 3.7443 | 0.34241 | 0.34241 | -3.79e+05 | RUSBoost | 239 | 0.18949 | 257 | 27 | deviance | - |
| 2 | Best | 0.22724 | 1.651 | 0.22724 | 0.23182 | -2.85e+05 | Bag | 101 | - | 47 | 218 | deviance | 1 |
| 3 | Best | 0.10739 | 0.47366 | 0.10739 | 0.1146 | -4.79e+05 | GentleBoost | 42 | 0.0062456 | 17 | 3 | - | - |
| 4 | Accept | 0.3214 | 0.55099 | 0.10739 | 0.11699 | -4.66e+05 | GentleBoost | 103 | 0.62239 | 408 | 1099 | - | - |
| 5 | Accept | 0.14864 | 0.31359 | 0.10739 | 0.11651 | -4.89e+05 | GentleBoost | 21 | 0.0053541 | 11 | 3 | - | - |
| 6 | Best | 0.074708 | 1.1135 | 0.074708 | 0.074725 | -4.03e+05 | GentleBoost | 196 | 0.0074455 | 34 | 3 | - | - |
| 7 | Best | 0.066148 | 1.27 | 0.066148 | 0.070734 | -3.86e+05 | GentleBoost | 230 | 0.011447 | 8 | 3 | - | - |
| 8 | Accept | 0.069261 | 2.3987 | 0.066148 | 0.066323 | -2.47e+05 | GentleBoost | 432 | 0.024744 | 4 | 4 | - | - |
| 9 | Accept | 0.12374 | 0.88609 | 0.066148 | 0.066281 | -4.32e+05 | GentleBoost | 168 | 0.16487 | 57 | 2 | - | - |
| 10 | Accept | 0.069261 | 1.0785 | 0.066148 | 0.070496 | -4.07e+05 | GentleBoost | 159 | 0.0012533 | 1 | 4 | - | - |
| 11 | Accept | 0.093385 | 1.5226 | 0.066148 | 0.066138 | -3.35e+05 | GentleBoost | 281 | 0.0020998 | 68 | 4 | - | - |
| 12 | Accept | 0.070817 | 1.0208 | 0.066148 | 0.066095 | -4.01e+05 | GentleBoost | 168 | 0.40426 | 2 | 4 | - | - |
| 13 | Best | 0.0607 | 2.2231 | 0.0607 | 0.060696 | -2.8e+05 | GentleBoost | 445 | 0.0061309 | 1 | 3 | - | - |
| 14 | Accept | 0.06537 | 1.1392 | 0.0607 | 0.060745 | -3.96e+05 | GentleBoost | 209 | 0.0038539 | 1 | 3 | - | - |
| 15 | Infeas | 0.067704 | 2.6118 | 0.0607 | 0.060826 | 4.43e+04 | GentleBoost | 448 | 0.045051 | 1 | 11 | - | - |
| 16 | Accept | 0.071595 | 0.82502 | 0.0607 | 0.061561 | -3.51e+05 | GentleBoost | 144 | 0.023625 | 1 | 9 | - | - |
| 17 | Accept | 0.067704 | 1.6121 | 0.0607 | 0.060871 | -2.7e+05 | GentleBoost | 301 | 0.014356 | 1 | 6 | - | - |
| 18 | Infeas | 0.081712 | 3.9201 | 0.0607 | 0.060855 | 1.56e+06 | Bag | 264 | - | 1 | 175 | deviance | 2 |
| 19 | Accept | 0.3393 | 0.18814 | 0.0607 | 0.060887 | -4.95e+05 | Bag | 12 | - | 38 | 1 | deviance | 2 |
| 20 | Infeas | 0.084047 | 5.6125 | 0.0607 | 0.060869 | 3e+06 | Bag | 405 | - | 1 | 121 | gdi | 1 |
|===============================================================================================================================================================================================|
| Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Constraint1 | Method | NumLearningC-| LearnRate | MinLeafSize | MaxNumSplits | SplitCriteri-| NumVariables-|
| | result | | runtime | (observed) | (estim.) | violation | | ycles | | | | on | ToSample |
|===============================================================================================================================================================================================|
| 21 | Accept | 0.28949 | 5.1019 | 0.0607 | 0.060906 | -2.03e+04 | Bag | 437 | - | 107 | 60 | gdi | 2 |
| 22 | Accept | 0.10895 | 2.4328 | 0.0607 | 0.062746 | -3.58e+05 | GentleBoost | 452 | 0.0019604 | 18 | 1 | - | - |
| 23 | Accept | 0.068482 | 1.055 | 0.0607 | 0.060844 | -3.84e+05 | GentleBoost | 171 | 0.0026648 | 5 | 5 | - | - |
| 24 | Accept | 0.066926 | 1.7971 | 0.0607 | 0.060832 | -2.29e+05 | GentleBoost | 262 | 0.86462 | 2 | 9 | - | - |
| 25 | Accept | 0.06537 | 1.4742 | 0.0607 | 0.06114 | -7.16e+04 | GentleBoost | 173 | 0.55591 | 1 | 25 | - | - |
| 26 | Accept | 0.062257 | 2.499 | 0.0607 | 0.060791 | -3.12e+05 | GentleBoost | 464 | 0.001241 | 1 | 2 | - | - |
| 27 | Accept | 0.066926 | 1.2121 | 0.0607 | 0.061207 | -4.04e+05 | GentleBoost | 194 | 0.0014837 | 5 | 3 | - | - |
| 28 | Accept | 0.064591 | 1.6711 | 0.0607 | 0.060801 | -1.98e+05 | GentleBoost | 216 | 0.1119 | 24 | 13 | - | - |
| 29 | Accept | 0.20311 | 1.0036 | 0.0607 | 0.061225 | -4.08e+05 | GentleBoost | 176 | 0.87764 | 223 | 27 | - | - |
| 30 | Infeas | 0.070039 | 3.058 | 0.0607 | 0.061289 | 2.28e+06 | GentleBoost | 256 | 0.38401 | 1 | 931 | - | - |
| 31 | Accept | 0.064591 | 0.24856 | 0.0607 | 0.061247 | -3.73e+05 | GentleBoost | 13 | 0.40922 | 1 | 228 | - | - |
| 32 | Accept | 0.070817 | 0.5699 | 0.0607 | 0.06139 | -9.93e+04 | GentleBoost | 38 | 0.0012624 | 1 | 126 | - | - |
| 33 | Accept | 0.063035 | 1.7115 | 0.0607 | 0.061101 | -2.43e+05 | GentleBoost | 300 | 0.0010972 | 17 | 7 | - | - |
| 34 | Accept | 0.068482 | 0.72834 | 0.0607 | 0.061114 | -7.9e+03 | GentleBoost | 52 | 0.65776 | 1 | 104 | - | - |
| 35 | Accept | 0.066926 | 0.18431 | 0.0607 | 0.061113 | -4.05e+05 | GentleBoost | 10 | 0.07542 | 1 | 704 | - | - |
| 36 | Accept | 0.067704 | 0.18884 | 0.0607 | 0.061083 | -3.96e+05 | GentleBoost | 13 | 0.031071 | 4 | 406 | - | - |
| 37 | Accept | 0.070817 | 0.67369 | 0.0607 | 0.061106 | -1.95e+05 | GentleBoost | 74 | 0.0016664 | 4 | 59 | - | - |
| 38 | Accept | 0.074708 | 0.27587 | 0.0607 | 0.061098 | -3.05e+05 | GentleBoost | 22 | 0.18784 | 2 | 464 | - | - |
| 39 | Accept | 0.10039 | 0.22573 | 0.0607 | 0.061088 | -4.14e+05 | Bag | 11 | - | 2 | 669 | deviance | 2 |
| 40 | Accept | 0.23813 | 0.27366 | 0.0607 | 0.061112 | -4.69e+05 | Bag | 14 | - | 46 | 1137 | deviance | 2 |
|===============================================================================================================================================================================================|
| Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Constraint1 | Method | NumLearningC-| LearnRate | MinLeafSize | MaxNumSplits | SplitCriteri-| NumVariables-|
| | result | | runtime | (observed) | (estim.) | violation | | ycles | | | | on | ToSample |
|===============================================================================================================================================================================================|
| 41 | Accept | 0.08716 | 0.41637 | 0.0607 | 0.061117 | -3.65e+05 | Bag | 22 | - | 1 | 58 | deviance | 2 |
| 42 | Accept | 0.080156 | 0.44531 | 0.0607 | 0.061237 | -3.3e+05 | Bag | 22 | - | 1 | 322 | deviance | 2 |
| 43 | Accept | 0.066148 | 0.9497 | 0.0607 | 0.061131 | -3.04e+05 | GentleBoost | 132 | 0.0010735 | 24 | 14 | - | - |
| 44 | Accept | 0.069261 | 0.17554 | 0.0607 | 0.061142 | -4.1e+05 | GentleBoost | 10 | 0.72738 | 2 | 155 | - | - |
| 45 | Accept | 0.080934 | 0.19364 | 0.0607 | 0.061135 | -3.84e+05 | LogitBoost | 14 | 0.069681 | 1 | 255 | - | - |
| 46 | Accept | 0.15875 | 0.17405 | 0.0607 | 0.061126 | -4.37e+05 | LogitBoost | 15 | 0.035786 | 13 | 217 | - | - |
| 47 | Accept | 0.14475 | 0.15086 | 0.0607 | 0.061121 | -4.83e+05 | LogitBoost | 13 | 0.44161 | 1 | 12 | - | - |
| 48 | Infeas | 0.066148 | 1.7258 | 0.0607 | 0.061125 | 1.38e+06 | LogitBoost | 186 | 0.12108 | 1 | 1235 | - | - |
| 49 | Accept | 0.084825 | 0.22487 | 0.0607 | 0.061131 | -4.11e+05 | LogitBoost | 11 | 0.053906 | 1 | 1260 | - | - |
| 50 | Accept | 0.44591 | 0.14745 | 0.0607 | 0.061293 | -4.96e+05 | LogitBoost | 10 | 0.0013377 | 465 | 1 | - | - |
__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 50 reached.
Total function evaluations: 50
Total elapsed time: 75.3564 seconds
Total objective function evaluation time: 65.1444
Best observed feasible point:
Method NumLearningCycles LearnRate MinLeafSize MaxNumSplits SplitCriterion NumVariablesToSample
___________ _________________ _________ ___________ ____________ ______________ ____________________
GentleBoost 445 0.0061309 1 3 <undefined> NaN
Observed objective function value = 0.0607
Estimated objective function value = 0.061293
Function evaluation time = 2.2231
Observed constraint violations =[ -279773.500000 ]
Best estimated feasible point (according to models):
Method NumLearningCycles LearnRate MinLeafSize MaxNumSplits SplitCriterion NumVariablesToSample
___________ _________________ _________ ___________ ____________ ______________ ____________________
GentleBoost 445 0.0061309 1 3 <undefined> NaN
Estimated objective function value = 0.061293
Estimated function evaluation time = 2.2238
Estimated constraint violations =[ -372671.500000 ]
optimizedSWClassifier = optimizedSWClassifier{1};Compute the 5-fold cross-validation accuracy.
cvOptimizedSWEnsemble = crossval(optimizedSWClassifier,KFold=5);
sprintf("Accuracy: %f",1 - kfoldLoss(cvOptimizedSWEnsemble))ans = "Accuracy: 0.922957"
The optimized model retains a cross-validation accuracy above 92%.
Compare Classifier Performance and Select Best Model
Compare the performance of the three shaft wear classifiers on the test data (faultDataTest and ShaftWearTest). Use the helper function helperComputeMetrics.
SWClassifiers = {defaultSWClassifier,smallSWClassifier, ...
optimizedSWClassifier};
ModelNames = ["Default model","Model with selected features", ...
"Optimized model with selected features"];
SWClassifiersMetricsTable = helperComputeMetrics(SWClassifiers, ...
faultDataTest,ShaftWearTest,ModelNames)SWClassifiersMetricsTable = 3×6 table
Accuracy True Positive Rate False Positive Rate False Positives False Negatives Model Size (KBs)
________ __________________ ___________________ _______________ _______________ ________________
Default model 0.93103 0.92245 0.062092 19 19 116.9
Model with selected features 0.96005 0.95102 0.03268 10 12 112.8
Optimized model with selected features 0.96003 0.94286 0.026144 8 14 220.23
The metrics table indicates that all three ensembles perform well on the test data.
Select smallSWClassifier as the shaft wear detector in the transmission system due to its high test set accuracy and its low deployment footprint.
SWClassifier = smallSWClassifier;
Reduce Size of Selected Classifiers
Reduce the size of the sensor drift classifier SDClassifier and the shaft wear classifier SWClassifier by using the compact function. The function removes properties, like the stored training data variables, that are not needed for making predictions on new data.
SDClassifier = compact(SDClassifier); SWClassifier = compact(SWClassifier);
Verify Model Fit Requirements
Verify that the final models SDClassifier and SWClassifier meet the requirements described in Model Fit Requirements. In particular, each classifier must perform well on the test data in terms of accuracy, true positive rate, and false positive rate, and must have a sufficiently low memory footprint. Use the helper function helperComputeMetrics to compute the necessary metrics. Use the helper function helperVerifyRequirements to summarize the results in a table.
sdMetrics = helperComputeMetrics({SDClassifier},faultDataTest,SensorDriftTest,"SD");
swMetrics = helperComputeMetrics({SWClassifier},faultDataTest,ShaftWearTest,"SW");
verificationTable = helperVerifyRequirements(sdMetrics,swMetrics)verificationTable = 8×5 table
RequirementID Description Metric Threshold Result
______________________ ______________________________________ ______ _________ ______
"SD_MODELFIT_ACCURACY" "Sensor drift detector accuracy > 92%" 99.819 92 "PASS"
"SW_MODELFIT_ACCURACY" "Shaft wear detector accuracy > 92%" 96.005 92 "PASS"
"SD_MODELFIT_TPR" "Sensor drift detector TPR > 90%" 100 90 "PASS"
"SW_MODELFIT_TPR" "Shaft wear detector TPR > 90%" 95.102 90 "PASS"
"SD_MODELFIT_FPR" "Sensor drift detector FPR < 5%" 0.3268 5 "PASS"
"SW_MODELFIT_FPR" "Shaft wear detector FPR < 5%" 3.268 5 "PASS"
"SD_MEMORY_FOOTPRINT" "Sensor drift model memory < 500 KB" 1.162 500 "PASS"
"SW_MEMORY_FOOTPRINT" "Shaft wear model memory < 500 KB" 112.8 500 "PASS"
The verification table shows that the classifiers pass all the requirements.
Helper Functions
helperComputeMetrics
The helperComputeMetrics helper function takes in a cell array of models (Mdls), predictor data (x), response data (y), and a string array of model names (MdlNames), and returns a table of metric values (metrics). In particular, for each model, the function computes the accuracy, true positive rate (TPR), false positive rate (FPR), number of false positives, and number of false negatives when predicting on the data in x and y. The function also estimates the amount of system memory the model uses in generated code (in kilobytes).
function metrics = helperComputeMetrics(Mdls,x,y,MdlNames) NumModels = numel(Mdls); % Initialize metrics Acc = zeros(NumModels,1); TPRs = Acc; FPRs = Acc; FPs = Acc; FNs = Acc; MdlSize = Acc; for i = 1:NumModels Mdl = Mdls{i}; Acc(i) = 1 - loss(Mdl,x,y); yPred = predict(Mdl,x); C = confusionmat(y,yPred,Order=[0 1]); % Extract counts % Rows = true class, Columns = predicted class TN = C(1,1); % true 0, predicted 0 FP = C(1,2); % true 0, predicted 1 FN = C(2,1); % true 1, predicted 0 TP = C(2,2); % true 1, predicted 1 % Compute TPR and FPR (guard against division by 0) TPR = TP / max(TP + FN,1); % Recall/Sensitivity for Positive=1 FPR = FP / max(FP + TN,1); % 1 - Specificity for Negative=0 TPRs(i) = TPR; FPRs(i) = FPR; FPs(i) = FP; FNs(i) = FN; MdlSize(i) = learnersize(Mdl,SizeType="coder")/1000; end metrics = table(Acc,TPRs,FPRs,FPs,FNs,MdlSize, ... VariableNames = ["Accuracy","True Positive Rate","False Positive Rate", ... "False Positives","False Negatives","Model Size (KBs)"], ... RowNames=MdlNames); end
helperVerifyRequirements
The helperVerifyRequirements function determines whether the requirements described in Model Fit Requirements are met, given the sensor drift classifier metrics sdMetrics and the shaft wear classifier metrics swMetrics. The function returns a table (verificationTable) with the results.
function verificationTable = helperVerifyRequirements(sdMetrics,swMetrics) RequirementID = ["SD_MODELFIT_ACCURACY";"SW_MODELFIT_ACCURACY"; ... "SD_MODELFIT_TPR";"SW_MODELFIT_TPR"; ... "SD_MODELFIT_FPR";"SW_MODELFIT_FPR"; ... "SD_MEMORY_FOOTPRINT";"SW_MEMORY_FOOTPRINT"]; Description = ["Sensor drift detector accuracy > 92%"; ... "Shaft wear detector accuracy > 92%"; ... "Sensor drift detector TPR > 90%"; ... "Shaft wear detector TPR > 90%"; ... "Sensor drift detector FPR < 5%"; ... "Shaft wear detector FPR < 5%"; ... "Sensor drift model memory < 500 KB"; ... "Shaft wear model memory < 500 KB"]; Metric = [sdMetrics.Accuracy*100; swMetrics.Accuracy*100; ... sdMetrics.("True Positive Rate")*100; swMetrics.("True Positive Rate")*100; ... sdMetrics.("False Positive Rate")*100; swMetrics.("False Positive Rate")*100; ... sdMetrics.("Model Size (KBs)"); swMetrics.("Model Size (KBs)")]; Threshold = [92; 92; 90; 90; 5; 5; 500; 500]; Pass = [Metric(1) > 92; Metric(2) > 92; Metric(3) > 90; Metric(4) > 90; ... Metric(5) < 5; Metric(6) < 5; Metric(7) < 500; Metric(8) < 500]; Result = strings(size(Pass)); Result(Pass) = "PASS"; Result(~Pass) = "FAIL"; verificationTable = table(RequirementID,Description,Metric,Threshold,Result); end
See Also
fitctree (Statistics and Machine Learning Toolbox) | fitcensemble (Statistics and Machine Learning Toolbox) | permutationImportance (Statistics and Machine Learning Toolbox) | crossval (Statistics and Machine Learning Toolbox) | compact (Statistics and Machine Learning Toolbox)
