Automated Classifier Selection with Bayesian and ASHA Optimization
This example shows how to use fitcauto to automatically try a selection of classification model types with different hyperparameter values, given training predictor and response data. By default, the function uses Bayesian optimization to select and assess models. If your training data set contains many observations, you can use an asynchronous successive halving algorithm (ASHA) instead. After the optimization is complete, fitcauto returns the model, trained on the entire data set, that is expected to best classify new data. Check the model performance on test data.
Load Sample Data
This example uses the 1994 census data stored in census1994.mat. The data set consists of demographic information from the US Census Bureau that can be used to predict whether an individual makes over $50,000 per year.
Load the sample data census1994, which contains the training data adultdata and the test data adulttest. Preview the first few rows of the training data set.
load census1994
head(adultdata) age workClass fnlwgt education education_num marital_status occupation relationship race sex capital_gain capital_loss hours_per_week native_country salary
___ ________________ __________ _________ _____________ _____________________ _________________ _____________ _____ ______ ____________ ____________ ______________ ______________ ______
39 State-gov 77516 Bachelors 13 Never-married Adm-clerical Not-in-family White Male 2174 0 40 United-States <=50K
50 Self-emp-not-inc 83311 Bachelors 13 Married-civ-spouse Exec-managerial Husband White Male 0 0 13 United-States <=50K
38 Private 2.1565e+05 HS-grad 9 Divorced Handlers-cleaners Not-in-family White Male 0 0 40 United-States <=50K
53 Private 2.3472e+05 11th 7 Married-civ-spouse Handlers-cleaners Husband Black Male 0 0 40 United-States <=50K
28 Private 3.3841e+05 Bachelors 13 Married-civ-spouse Prof-specialty Wife Black Female 0 0 40 Cuba <=50K
37 Private 2.8458e+05 Masters 14 Married-civ-spouse Exec-managerial Wife White Female 0 0 40 United-States <=50K
49 Private 1.6019e+05 9th 5 Married-spouse-absent Other-service Not-in-family Black Female 0 0 16 Jamaica <=50K
52 Self-emp-not-inc 2.0964e+05 HS-grad 9 Married-civ-spouse Exec-managerial Husband White Male 0 0 45 United-States >50K
Each row contains the demographic information for one adult. The last column salary shows whether a person has a salary less than or equal to $50,000 per year or greater than $50,000 per year.
Remove observations from adultdata and adulttest that contain missing values.
adultdata = rmmissing(adultdata); adulttest = rmmissing(adulttest);
Use Automated Model Selection with Bayesian Optimization
Find an appropriate classifier for the data in adultdata by using fitcauto. By default, fitcauto uses Bayesian optimization to select models and their hyperparameter values, and computes the cross-validation classification error (Validation loss) for each model. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
Set the observation weights, and specify to run the Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. Because of the complexity of the optimization, this process can take some time, especially for larger data sets.
bayesianOptions = struct("UseParallel",true); [bayesianMdl,bayesianResults] = fitcauto(adultdata,"salary","Weights","fnlwgt", ... "HyperparameterOptimizationOptions",bayesianOptions);
Warning: Data set has more than 10000 observations. Because ASHA optimization often finds good solutions faster than Bayesian optimization for data sets with many observations, try specifying the 'Optimizer' field value as 'asha' in the 'HyperparameterOptimizationOptions' value structure.
Starting parallel pool (parpool) using the 'Processes' profile ... Connected to parallel pool with 6 workers. Copying objective function to workers... Done copying objective function to workers. Learner types to explore: ensemble, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 120 Total time (MaxTime): Inf |=======================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |=======================================================================================================================================================| | 1 | 6 | Best | 0.24677 | 43.754 | 0.24677 | 0.24677 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 207 | | | | | | | | | | MinLeafSize: 8946 | | 2 | 6 | Accept | 0.24677 | 0.88853 | 0.24677 | 0.24677 | tree | MinLeafSize: 6845 | | 3 | 6 | Best | 0.14543 | 2.2401 | 0.14543 | 0.19002 | tree | MinLeafSize: 45 | | 4 | 6 | Accept | 0.15419 | 80.045 | 0.14543 | 0.19002 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 209 | | | | | | | | | | MinLeafSize: 2 | | 5 | 6 | Accept | 0.14928 | 93.473 | 0.14543 | 0.18783 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 201 | | | | | | | | | | MinLeafSize: 480 | | 6 | 6 | Accept | 0.15158 | 44.195 | 0.14543 | 0.15158 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0088611 | | | | | | | | | | Standardize: false | | 7 | 6 | Accept | 0.14889 | 139.61 | 0.14543 | 0.15158 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 297 | | | | | | | | | | MinLeafSize: 510 | | 8 | 6 | Accept | 0.18439 | 1.4145 | 0.14543 | 0.15158 | tree | MinLeafSize: 3104 | | 9 | 6 | Accept | 0.14554 | 161.4 | 0.14543 | 0.14554 | svm | BoxConstraint: 0.65272 | | | | | | | | | | KernelScale: 6.0716 | | 10 | 6 | Accept | 0.15127 | 166.83 | 0.14543 | 0.14881 | svm | BoxConstraint: 0.29711 | | | | | | | | | | KernelScale: 12.637 | | 11 | 6 | Accept | 0.16715 | 76.475 | 0.14543 | 0.14881 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.22835 | | | | | | | | | | Standardize: true | | 12 | 6 | Accept | 0.20124 | 101.74 | 0.14543 | 0.14881 | nb | DistributionNames: kernel | | | | | | | | | | Width: 18.24 | | | | | | | | | | Standardize: true | | 13 | 6 | Accept | 0.24677 | 0.46731 | 0.14543 | 0.14881 | tree | MinLeafSize: 12859 | | 14 | 6 | Accept | 0.168 | 0.69093 | 0.14543 | 0.14881 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 15 | 6 | Accept | 0.24677 | 3.7861 | 0.14543 | 0.17365 | svm | BoxConstraint: 0.24765 | | | | | | | | | | KernelScale: 0.0016358 | | 16 | 6 | Accept | 0.14757 | 1.8122 | 0.14543 | 0.17365 | tree | MinLeafSize: 135 | | 17 | 6 | Accept | 0.168 | 0.36422 | 0.14543 | 0.17099 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 18 | 6 | Accept | 0.23488 | 55.736 | 0.14543 | 0.17099 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 243 | | | | | | | | | | MinLeafSize: 5284 | | 19 | 6 | Accept | 0.24677 | 48.557 | 0.14543 | 0.17099 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 217 | | | | | | | | | | MinLeafSize: 6309 | | 20 | 6 | Accept | 0.16312 | 44.074 | 0.14543 | 0.16836 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.043878 | | | | | | | | | | Standardize: true | |=======================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |=======================================================================================================================================================| | 21 | 6 | Accept | 0.19857 | 99.234 | 0.14543 | 0.17104 | nb | DistributionNames: kernel | | | | | | | | | | Width: 3.1517 | | | | | | | | | | Standardize: true | | 22 | 6 | Accept | 0.15361 | 1.5262 | 0.14543 | 0.16123 | tree | MinLeafSize: 338 | | 23 | 6 | Accept | 0.15417 | 92.662 | 0.14543 | 0.16123 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 277 | | | | | | | | | | MinLeafSize: 101 | | 24 | 6 | Accept | 0.1628 | 2.9924 | 0.14543 | 0.15776 | tree | MinLeafSize: 6 | | 25 | 6 | Accept | 0.15479 | 85.937 | 0.14543 | 0.15776 | nb | DistributionNames: kernel | | | | | | | | | | Width: 4.0678 | | | | | | | | | | Standardize: false | | 26 | 6 | Best | 0.1445 | 1.7904 | 0.1445 | 0.15547 | tree | MinLeafSize: 105 | | 27 | 6 | Accept | 0.15763 | 2.8146 | 0.1445 | 0.15223 | tree | MinLeafSize: 9 | | 28 | 6 | Accept | 0.24677 | 1.9596 | 0.1445 | 0.15223 | svm | BoxConstraint: 0.0047884 | | | | | | | | | | KernelScale: 0.0048876 | | 29 | 6 | Accept | 0.15158 | 42.575 | 0.1445 | 0.15223 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0075341 | | | | | | | | | | Standardize: false | | 30 | 6 | Accept | 0.16813 | 1.1328 | 0.1445 | 0.15262 | tree | MinLeafSize: 598 | | 31 | 6 | Accept | 0.14994 | 2.2018 | 0.1445 | 0.15084 | tree | MinLeafSize: 21 | | 32 | 6 | Accept | 0.168 | 0.4177 | 0.1445 | 0.15084 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 33 | 6 | Accept | 0.14535 | 165.79 | 0.1445 | 0.15084 | svm | BoxConstraint: 0.14315 | | | | | | | | | | KernelScale: 3.2371 | | 34 | 6 | Accept | 0.17901 | 5.6933 | 0.1445 | 0.1502 | tree | MinLeafSize: 2 | | 35 | 6 | Accept | 0.168 | 0.48234 | 0.1445 | 0.1502 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 36 | 6 | Accept | 0.16795 | 1.0938 | 0.1445 | 0.15088 | tree | MinLeafSize: 570 | | 37 | 6 | Accept | 0.18403 | 0.65343 | 0.1445 | 0.15009 | tree | MinLeafSize: 2119 | | 38 | 6 | Accept | 0.24677 | 2.0224 | 0.1445 | 0.15009 | svm | BoxConstraint: 216.63 | | | | | | | | | | KernelScale: 0.006762 | | 39 | 6 | Accept | 0.14945 | 83.448 | 0.1445 | 0.15009 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 223 | | | | | | | | | | MinLeafSize: 345 | | 40 | 6 | Accept | 0.16129 | 90.614 | 0.1445 | 0.15009 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 278 | | | | | | | | | | MinLeafSize: 388 | |=======================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |=======================================================================================================================================================| | 41 | 6 | Accept | 0.1647 | 32.478 | 0.1445 | 0.15009 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 256 | | | | | | | | | | MinLeafSize: 7032 | | 42 | 6 | Accept | 0.16698 | 71.592 | 0.1445 | 0.15009 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 226 | | | | | | | | | | MinLeafSize: 861 | | 43 | 6 | Accept | 0.15872 | 189.88 | 0.1445 | 0.15009 | svm | BoxConstraint: 11.079 | | | | | | | | | | KernelScale: 155.11 | | 44 | 6 | Best | 0.14353 | 1.687 | 0.14353 | 0.14816 | tree | MinLeafSize: 79 | | 45 | 6 | Accept | 0.15374 | 39.891 | 0.14353 | 0.14816 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0035642 | | | | | | | | | | Standardize: true | | 46 | 6 | Accept | 0.14979 | 83.79 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 203 | | | | | | | | | | MinLeafSize: 911 | | 47 | 6 | Accept | 0.24677 | 2.0222 | 0.14353 | 0.14816 | svm | BoxConstraint: 2.0475 | | | | | | | | | | KernelScale: 0.0049311 | | 48 | 6 | Accept | 0.20139 | 94.777 | 0.14353 | 0.14816 | nb | DistributionNames: kernel | | | | | | | | | | Width: 93.844 | | | | | | | | | | Standardize: true | | 49 | 6 | Accept | 0.168 | 0.38418 | 0.14353 | 0.14816 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 50 | 6 | Accept | 0.15146 | 47.829 | 0.14353 | 0.14816 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.36089 | | | | | | | | | | Standardize: false | | 51 | 6 | Accept | 0.1686 | 27.403 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 261 | | | | | | | | | | MinLeafSize: 8358 | | 52 | 6 | Accept | 0.23867 | 195.4 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.011694 | | | | | | | | | | KernelScale: 18.422 | | 53 | 6 | Accept | 0.15319 | 162.54 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.010059 | | | | | | | | | | KernelScale: 4.0044 | | 54 | 6 | Accept | 0.15501 | 79.775 | 0.14353 | 0.14816 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 232 | | | | | | | | | | MinLeafSize: 24 | | 55 | 6 | Accept | 0.16874 | 2864.2 | 0.14353 | 0.14816 | svm | BoxConstraint: 2.3437 | | | | | | | | | | KernelScale: 1.5289 | | 56 | 6 | Accept | 0.15742 | 2870.4 | 0.14353 | 0.14816 | svm | BoxConstraint: 586.48 | | | | | | | | | | KernelScale: 6.2332 | | 57 | 6 | Accept | 0.15119 | 81.33 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 209 | | | | | | | | | | MinLeafSize: 1622 | | 58 | 6 | Accept | 0.16761 | 3059.5 | 0.14353 | 0.14816 | svm | BoxConstraint: 380.05 | | | | | | | | | | KernelScale: 3.8304 | | 59 | 6 | Accept | 0.1522 | 77.374 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 204 | | | | | | | | | | MinLeafSize: 1676 | | 60 | 6 | Accept | 0.17435 | 3048.3 | 0.14353 | 0.14816 | svm | BoxConstraint: 1.424 | | | | | | | | | | KernelScale: 1.1302 | |=======================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |=======================================================================================================================================================| | 61 | 6 | Accept | 0.15944 | 62.648 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 228 | | | | | | | | | | MinLeafSize: 2945 | | 62 | 6 | Accept | 0.1622 | 2716.5 | 0.14353 | 0.14816 | svm | BoxConstraint: 8.1184 | | | | | | | | | | KernelScale: 2.3874 | | 63 | 6 | Accept | 0.15986 | 53.967 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 203 | | | | | | | | | | MinLeafSize: 3145 | | 64 | 6 | Accept | 0.14553 | 150.62 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.26261 | | | | | | | | | | KernelScale: 5.0699 | | 65 | 6 | Accept | 0.14799 | 157.27 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.081785 | | | | | | | | | | KernelScale: 4.9516 | | 66 | 6 | Accept | 0.14833 | 158.24 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.74137 | | | | | | | | | | KernelScale: 12.172 | | 67 | 6 | Accept | 0.20535 | 205.52 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.26823 | | | | | | | | | | KernelScale: 60.591 | | 68 | 6 | Accept | 0.15533 | 173.46 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.031445 | | | | | | | | | | KernelScale: 7.4327 | | 69 | 6 | Accept | 0.16052 | 179.49 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.0062986 | | | | | | | | | | KernelScale: 4.855 | | 70 | 6 | Accept | 0.16085 | 46.246 | 0.14353 | 0.14816 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 211 | | | | | | | | | | MinLeafSize: 3991 | | 71 | 6 | Accept | 0.15032 | 159.22 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.34417 | | | | | | | | | | KernelScale: 12.342 | | 72 | 6 | Accept | 0.18532 | 46.838 | 0.14353 | 0.14816 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 202 | | | | | | | | | | MinLeafSize: 4515 | | 73 | 6 | Accept | 0.18875 | 189.42 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.0019135 | | | | | | | | | | KernelScale: 5.2624 | | 74 | 6 | Accept | 0.15372 | 161.15 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.2716 | | | | | | | | | | KernelScale: 15.245 | | 75 | 6 | Accept | 0.15399 | 159.07 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.01694 | | | | | | | | | | KernelScale: 4.9823 | | 76 | 6 | Accept | 0.15324 | 160.44 | 0.14353 | 0.1477 | svm | BoxConstraint: 0.35465 | | | | | | | | | | KernelScale: 16.266 | | 77 | 6 | Accept | 0.14811 | 149.74 | 0.14353 | 0.1474 | svm | BoxConstraint: 0.13086 | | | | | | | | | | KernelScale: 5.7377 | | 78 | 6 | Accept | 0.15523 | 165.32 | 0.14353 | 0.14769 | svm | BoxConstraint: 0.18542 | | | | | | | | | | KernelScale: 15.367 | | 79 | 6 | Accept | 0.15441 | 160.44 | 0.14353 | 0.14811 | svm | BoxConstraint: 0.025242 | | | | | | | | | | KernelScale: 5.8741 | | 80 | 6 | Accept | 0.15792 | 169.29 | 0.14353 | 0.14816 | svm | BoxConstraint: 0.013558 | | | | | | | | | | KernelScale: 5.9968 | |=======================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |=======================================================================================================================================================| | 81 | 6 | Accept | 0.14914 | 151.25 | 0.14353 | 0.14816 | svm | BoxConstraint: 1.0545 | | | | | | | | | | KernelScale: 15.459 | | 82 | 6 | Accept | 0.18184 | 48.24 | 0.14353 | 0.14816 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 205 | | | | | | | | | | MinLeafSize: 4097 | | 83 | 6 | Accept | 0.14507 | 164.62 | 0.14353 | 0.14715 | svm | BoxConstraint: 0.29922 | | | | | | | | | | KernelScale: 4.0302 | | 84 | 6 | Accept | 0.15297 | 159.83 | 0.14353 | 0.14669 | svm | BoxConstraint: 0.42207 | | | | | | | | | | KernelScale: 17.016 | | 85 | 6 | Accept | 0.14538 | 164.56 | 0.14353 | 0.14584 | svm | BoxConstraint: 0.30157 | | | | | | | | | | KernelScale: 3.9176 | | 86 | 6 | Accept | 0.15171 | 146.64 | 0.14353 | 0.14681 | svm | BoxConstraint: 0.011941 | | | | | | | | | | KernelScale: 3.7945 | | 87 | 6 | Accept | 0.15038 | 159.26 | 0.14353 | 0.14624 | svm | BoxConstraint: 0.8291 | | | | | | | | | | KernelScale: 17.344 | | 88 | 6 | Accept | 0.14811 | 145.43 | 0.14353 | 0.14638 | svm | BoxConstraint: 2.3513 | | | | | | | | | | KernelScale: 17.792 | | 89 | 6 | Accept | 0.14957 | 145.61 | 0.14353 | 0.14614 | svm | BoxConstraint: 1.2721 | | | | | | | | | | KernelScale: 17.942 | | 90 | 6 | Accept | 0.43333 | 3817 | 0.14353 | 0.14739 | svm | BoxConstraint: 0.46095 | | | | | | | | | | KernelScale: 0.11266 | | 91 | 6 | Accept | 0.14777 | 148.7 | 0.14353 | 0.14651 | svm | BoxConstraint: 3.0283 | | | | | | | | | | KernelScale: 18.144 | | 92 | 6 | Accept | 0.37513 | 2462.6 | 0.14353 | 0.14633 | svm | BoxConstraint: 0.15218 | | | | | | | | | | KernelScale: 0.15693 | | 93 | 6 | Accept | 0.37822 | 2568 | 0.14353 | 0.14674 | svm | BoxConstraint: 0.085986 | | | | | | | | | | KernelScale: 0.15131 | | 94 | 6 | Accept | 0.31891 | 2559 | 0.14353 | 0.14582 | svm | BoxConstraint: 0.142 | | | | | | | | | | KernelScale: 0.14667 | | 95 | 6 | Accept | 0.35037 | 2484.8 | 0.14353 | 0.14719 | svm | BoxConstraint: 732.35 | | | | | | | | | | KernelScale: 0.30475 | | 96 | 6 | Accept | 0.20831 | 3290.1 | 0.14353 | 0.14655 | svm | BoxConstraint: 0.36082 | | | | | | | | | | KernelScale: 0.55221 | | 97 | 6 | Accept | 0.24678 | 4132.6 | 0.14353 | 0.14626 | svm | BoxConstraint: 0.24182 | | | | | | | | | | KernelScale: 0.032229 | | 98 | 6 | Accept | 0.21716 | 3318.5 | 0.14353 | 0.14621 | svm | BoxConstraint: 0.66883 | | | | | | | | | | KernelScale: 0.59271 | | 99 | 6 | Accept | 0.14578 | 248.04 | 0.14353 | 0.14582 | svm | BoxConstraint: 0.4278 | | | | | | | | | | KernelScale: 3.0737 | | 100 | 6 | Accept | 0.15224 | 162.91 | 0.14353 | 0.14686 | svm | BoxConstraint: 1.1624 | | | | | | | | | | KernelScale: 24.402 | |=======================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |=======================================================================================================================================================| | 101 | 6 | Accept | 0.22215 | 3344 | 0.14353 | 0.14576 | svm | BoxConstraint: 0.17981 | | | | | | | | | | KernelScale: 0.41659 | | 102 | 6 | Accept | 0.14999 | 160.35 | 0.14353 | 0.1458 | svm | BoxConstraint: 1.1293 | | | | | | | | | | KernelScale: 18.197 | | 103 | 6 | Accept | 0.14671 | 265.17 | 0.14353 | 0.14599 | svm | BoxConstraint: 0.29559 | | | | | | | | | | KernelScale: 2.7758 | | 104 | 6 | Accept | 0.17255 | 191.76 | 0.14353 | 0.14571 | svm | BoxConstraint: 0.055759 | | | | | | | | | | KernelScale: 18.892 | | 105 | 6 | Accept | 0.15195 | 158.78 | 0.14353 | 0.14564 | svm | BoxConstraint: 0.012192 | | | | | | | | | | KernelScale: 3.9397 | | 106 | 6 | Accept | 0.16752 | 186.75 | 0.14353 | 0.14569 | svm | BoxConstraint: 0.074389 | | | | | | | | | | KernelScale: 18.359 | | 107 | 6 | Accept | 0.14911 | 153.52 | 0.14353 | 0.14596 | svm | BoxConstraint: 0.011498 | | | | | | | | | | KernelScale: 3.1239 | | 108 | 6 | Accept | 0.22432 | 3370.4 | 0.14353 | 0.14617 | svm | BoxConstraint: 0.26224 | | | | | | | | | | KernelScale: 0.48333 | | 109 | 6 | Accept | 0.14851 | 150.42 | 0.14353 | 0.14601 | svm | BoxConstraint: 0.011062 | | | | | | | | | | KernelScale: 2.9993 | | 110 | 6 | Accept | 0.14605 | 302.86 | 0.14353 | 0.14605 | svm | BoxConstraint: 1.2847 | | | | | | | | | | KernelScale: 3.6109 | | 111 | 6 | Accept | 0.14905 | 479.09 | 0.14353 | 0.14572 | svm | BoxConstraint: 0.37863 | | | | | | | | | | KernelScale: 2.2975 | | 112 | 6 | Accept | 0.17238 | 188.74 | 0.14353 | 0.14591 | svm | BoxConstraint: 0.011768 | | | | | | | | | | KernelScale: 9.0656 | | 113 | 6 | Accept | 0.14986 | 154.37 | 0.14353 | 0.1458 | svm | BoxConstraint: 0.0091472 | | | | | | | | | | KernelScale: 3.0854 | | 114 | 6 | Accept | 0.14945 | 150.17 | 0.14353 | 0.1456 | svm | BoxConstraint: 0.010472 | | | | | | | | | | KernelScale: 3.1579 | | 115 | 6 | Accept | 0.14585 | 336.82 | 0.14353 | 0.14577 | svm | BoxConstraint: 2.477 | | | | | | | | | | KernelScale: 4.0884 | | 116 | 6 | Accept | 0.14579 | 226.31 | 0.14353 | 0.14583 | svm | BoxConstraint: 0.19394 | | | | | | | | | | KernelScale: 2.6603 | | 117 | 6 | Accept | 0.15424 | 157.96 | 0.14353 | 0.14587 | svm | BoxConstraint: 0.33622 | | | | | | | | | | KernelScale: 18.198 | | 118 | 6 | Accept | 0.14706 | 311.23 | 0.14353 | 0.14599 | svm | BoxConstraint: 0.27404 | | | | | | | | | | KernelScale: 2.4769 | | 119 | 6 | Accept | 0.15064 | 576.64 | 0.14353 | 0.1456 | svm | BoxConstraint: 0.20057 | | | | | | | | | | KernelScale: 1.9187 | | 120 | 6 | Accept | 0.22663 | 3502.9 | 0.14353 | 0.14568 | svm | BoxConstraint: 26.696 | | | | | | | | | | KernelScale: 0.83468 |
__________________________________________________________ Optimization completed. Total iterations: 120 Total elapsed time: 11529.2732 seconds Total time for training and validation: 61054.1309 seconds Best observed learner is a tree model with: Learner: tree MinLeafSize: 79 Observed validation loss: 0.14353 Time for training and validation: 1.687 seconds Best estimated learner (returned model) is an svm model with: Learner: svm BoxConstraint: 0.29922 KernelScale: 4.0302 Estimated validation loss: 0.14568 Estimated time for training and validation: 168.4716 seconds Documentation for fitcauto display
The Total elapsed time value shows that the Bayesian optimization took a while to run (about 3.2 hours).
The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data set (adultdata), the listed Learner (or model) type, and the displayed hyperparameter values.
Use Automated Model Selection with ASHA Optimization
When fitcauto with Bayesian optimization takes a long time to run because of the number of observations in your training set, consider using fitcauto with ASHA optimization instead. Given that adultdata contains over 10,000 observations, try using fitcauto with ASHA optimization to automatically find an appropriate classifier. When you use fitcauto with ASHA optimization, the function randomly chooses several models with different hyperparameter values and trains them on a small subset of the training data. If the cross-validation classification error (Validation Loss) of a particular model is promising, the model is promoted and trained on a larger amount of the training data. This process repeats, and successful models are trained on progressively larger amounts of data. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
Set the observation weights, and specify to run the ASHA optimization in parallel. Note that ASHA optimization often has more iterations than Bayesian optimization by default. If you have a time constraint, you can specify the MaxTime field of the HyperparameterOptimizationOptions structure to limit the number of seconds fitcauto runs.
ashaOptions = struct("Optimizer","asha","UseParallel",true); [ashaMdl,ashaResults] = fitcauto(adultdata,"salary","Weights","fnlwgt", ... "HyperparameterOptimizationOptions",ashaOptions);
Copying objective function to workers...
Warning: Files that have already been attached are being ignored. To see which files are attached see the 'AttachedFiles' property of the parallel pool.
Done copying objective function to workers. Learner types to explore: ensemble, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 425 Total time (MaxTime): Inf |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 1 | 6 | Best | 0.24677 | 0.41223 | 0.24677 | 378 | tree | MinLeafSize: 293 | | 2 | 5 | Best | 0.21242 | 0.69352 | 0.21242 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 3 | 5 | Accept | 0.24677 | 0.49992 | 0.21242 | 378 | tree | MinLeafSize: 1192 | | 4 | 5 | Accept | 0.24677 | 0.89242 | 0.21242 | 378 | tree | MinLeafSize: 10141 | | 5 | 5 | Best | 0.18925 | 0.26458 | 0.18925 | 378 | tree | MinLeafSize: 16 | | 6 | 6 | Best | 0.18343 | 0.41943 | 0.18343 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 7 | 6 | Accept | 0.24677 | 0.98959 | 0.18343 | 378 | svm | BoxConstraint: 1.3947 | | | | | | | | | | KernelScale: 65.678 | | 8 | 6 | Accept | 0.18372 | 0.53536 | 0.18343 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 9 | 6 | Accept | 0.1954 | 9.5036 | 0.18343 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0034879 | | | | | | | | | | Standardize: true | | 10 | 6 | Best | 0.1792 | 0.5341 | 0.1792 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 11 | 6 | Best | 0.17449 | 0.425 | 0.17449 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 12 | 6 | Accept | 0.24677 | 14.424 | 0.17449 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 295 | | | | | | | | | | MinLeafSize: 4119 | | 13 | 6 | Accept | 0.20024 | 0.42798 | 0.17449 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 14 | 6 | Accept | 0.20304 | 13.33 | 0.17449 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 2.8059 | | | | | | | | | | Standardize: true | | 15 | 6 | Accept | 0.19474 | 14.061 | 0.17449 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 25.229 | | | | | | | | | | Standardize: true | | 16 | 6 | Best | 0.16052 | 0.3778 | 0.16052 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 17 | 6 | Accept | 0.20451 | 0.20402 | 0.16052 | 378 | tree | MinLeafSize: 19 | | 18 | 6 | Accept | 0.24677 | 0.1917 | 0.16052 | 378 | tree | MinLeafSize: 1648 | | 19 | 6 | Accept | 0.182 | 0.48008 | 0.16052 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 20 | 6 | Accept | 0.21976 | 0.84268 | 0.16052 | 378 | svm | BoxConstraint: 1.374 | | | | | | | | | | KernelScale: 1.1138 | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 21 | 6 | Accept | 0.16445 | 0.40264 | 0.16052 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 22 | 6 | Accept | 0.17111 | 0.39243 | 0.16052 | 6033 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 23 | 6 | Accept | 0.17395 | 0.87746 | 0.16052 | 378 | svm | BoxConstraint: 1.3291 | | | | | | | | | | KernelScale: 11.5 | | 24 | 6 | Accept | 0.24677 | 0.88712 | 0.16052 | 378 | svm | BoxConstraint: 2.32 | | | | | | | | | | KernelScale: 249.93 | | 25 | 6 | Accept | 0.18863 | 8.5476 | 0.16052 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.050938 | | | | | | | | | | Standardize: true | | 26 | 6 | Accept | 0.39894 | 19.121 | 0.16052 | 378 | svm | BoxConstraint: 0.082795 | | | | | | | | | | KernelScale: 0.11799 | | 27 | 6 | Accept | 0.21388 | 0.84483 | 0.16052 | 378 | svm | BoxConstraint: 536.31 | | | | | | | | | | KernelScale: 2.96 | | 28 | 6 | Accept | 0.17673 | 0.40821 | 0.16052 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 29 | 6 | Accept | 0.24677 | 0.14927 | 0.16052 | 378 | tree | MinLeafSize: 228 | | 30 | 6 | Best | 0.15579 | 2.158 | 0.15579 | 1509 | svm | BoxConstraint: 1.3291 | | | | | | | | | | KernelScale: 11.5 | | 31 | 6 | Accept | 0.16985 | 0.38832 | 0.15579 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 32 | 6 | Accept | 0.24677 | 0.30699 | 0.15579 | 378 | svm | BoxConstraint: 0.045199 | | | | | | | | | | KernelScale: 0.0063948 | | 33 | 6 | Accept | 0.20589 | 0.20066 | 0.15579 | 378 | tree | MinLeafSize: 22 | | 34 | 6 | Accept | 0.24677 | 0.90412 | 0.15579 | 378 | svm | BoxConstraint: 0.059749 | | | | | | | | | | KernelScale: 179.59 | | 35 | 6 | Accept | 0.17167 | 11.632 | 0.15579 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.3336 | | | | | | | | | | Standardize: true | | 36 | 6 | Accept | 0.22055 | 12.98 | 0.15579 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 12.182 | | | | | | | | | | Standardize: true | | 37 | 6 | Accept | 0.16867 | 12.841 | 0.15579 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 108.9 | | | | | | | | | | Standardize: false | | 38 | 6 | Accept | 0.16266 | 0.86177 | 0.15579 | 378 | svm | BoxConstraint: 385.35 | | | | | | | | | | KernelScale: 65.814 | | 39 | 6 | Accept | 0.22491 | 0.8454 | 0.15579 | 378 | svm | BoxConstraint: 11.832 | | | | | | | | | | KernelScale: 1.7245 | | 40 | 6 | Accept | 0.1561 | 2.1629 | 0.15579 | 1509 | svm | BoxConstraint: 385.35 | | | | | | | | | | KernelScale: 65.814 | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 41 | 6 | Accept | 0.22074 | 0.28317 | 0.15579 | 378 | tree | MinLeafSize: 22 | | 42 | 6 | Accept | 0.24677 | 12.744 | 0.15579 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 297 | | | | | | | | | | MinLeafSize: 7579 | | 43 | 6 | Accept | 0.24677 | 14.118 | 0.15579 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 282 | | | | | | | | | | MinLeafSize: 1543 | | 44 | 6 | Accept | 0.16998 | 0.33582 | 0.15579 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 45 | 6 | Accept | 0.31094 | 37.128 | 0.15579 | 378 | svm | BoxConstraint: 70.951 | | | | | | | | | | KernelScale: 0.033775 | | 46 | 6 | Accept | 0.24677 | 0.34815 | 0.15579 | 378 | svm | BoxConstraint: 623.48 | | | | | | | | | | KernelScale: 0.0010192 | | 47 | 6 | Accept | 0.17045 | 10.999 | 0.15579 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.8198 | | | | | | | | | | Standardize: false | | 48 | 6 | Accept | 0.15807 | 13.413 | 0.15579 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 220 | | | | | | | | | | MinLeafSize: 30 | | 49 | 6 | Accept | 0.17396 | 29.339 | 0.15579 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.3336 | | | | | | | | | | Standardize: true | | 50 | 6 | Accept | 0.676 | 23.204 | 0.15579 | 378 | svm | BoxConstraint: 39.423 | | | | | | | | | | KernelScale: 0.10596 | | 51 | 6 | Accept | 0.20495 | 0.22109 | 0.15579 | 378 | tree | MinLeafSize: 21 | | 52 | 6 | Accept | 0.18563 | 0.20139 | 0.15579 | 378 | tree | MinLeafSize: 42 | | 53 | 6 | Accept | 0.24677 | 0.92178 | 0.15579 | 378 | svm | BoxConstraint: 0.05188 | | | | | | | | | | KernelScale: 44.452 | | 54 | 6 | Accept | 0.18321 | 0.38725 | 0.15579 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 55 | 6 | Accept | 0.17723 | 0.4152 | 0.15579 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 56 | 6 | Accept | 0.48003 | 26.08 | 0.15579 | 378 | svm | BoxConstraint: 213.45 | | | | | | | | | | KernelScale: 0.092072 | | 57 | 6 | Best | 0.15393 | 17.242 | 0.15393 | 1509 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 220 | | | | | | | | | | MinLeafSize: 30 | | 58 | 6 | Accept | 0.17298 | 0.89838 | 0.15393 | 378 | svm | BoxConstraint: 962.26 | | | | | | | | | | KernelScale: 207.85 | | 59 | 6 | Best | 0.153 | 12.348 | 0.153 | 6033 | svm | BoxConstraint: 1.3291 | | | | | | | | | | KernelScale: 11.5 | | 60 | 6 | Accept | 0.18192 | 0.89064 | 0.153 | 378 | svm | BoxConstraint: 0.0055891 | | | | | | | | | | KernelScale: 0.90898 | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 61 | 6 | Accept | 0.18271 | 32.278 | 0.153 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 108.9 | | | | | | | | | | Standardize: false | | 62 | 6 | Accept | 0.24677 | 0.14112 | 0.153 | 378 | tree | MinLeafSize: 3783 | | 63 | 6 | Accept | 0.26476 | 34.581 | 0.153 | 378 | svm | BoxConstraint: 74.583 | | | | | | | | | | KernelScale: 0.059475 | | 64 | 6 | Accept | 0.20762 | 8.8087 | 0.153 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.10297 | | | | | | | | | | Standardize: false | | 65 | 6 | Accept | 0.24677 | 1.0071 | 0.153 | 378 | svm | BoxConstraint: 0.09297 | | | | | | | | | | KernelScale: 536.28 | | 66 | 6 | Accept | 0.24194 | 1.071 | 0.153 | 378 | svm | BoxConstraint: 0.67333 | | | | | | | | | | KernelScale: 35.903 | | 67 | 6 | Accept | 0.20385 | 18.897 | 0.153 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 252 | | | | | | | | | | MinLeafSize: 77 | | 68 | 6 | Accept | 0.15503 | 2.2706 | 0.153 | 1509 | svm | BoxConstraint: 962.26 | | | | | | | | | | KernelScale: 207.85 | | 69 | 6 | Accept | 0.24677 | 13.432 | 0.153 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 245 | | | | | | | | | | MinLeafSize: 3552 | | 70 | 6 | Accept | 0.19077 | 8.1226 | 0.153 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.027553 | | | | | | | | | | Standardize: false | | 71 | 6 | Accept | 0.16757 | 0.36842 | 0.153 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 72 | 6 | Accept | 0.18779 | 0.35808 | 0.153 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 73 | 6 | Accept | 0.17793 | 12.899 | 0.153 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 21.131 | | | | | | | | | | Standardize: false | | 74 | 6 | Accept | 0.16496 | 27.002 | 0.153 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.8198 | | | | | | | | | | Standardize: false | | 75 | 6 | Accept | 0.24677 | 0.21047 | 0.153 | 378 | tree | MinLeafSize: 12519 | | 76 | 6 | Accept | 0.16784 | 0.90871 | 0.153 | 378 | svm | BoxConstraint: 763.29 | | | | | | | | | | KernelScale: 144.74 | | 77 | 6 | Accept | 0.24677 | 8.515 | 0.153 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 210 | | | | | | | | | | MinLeafSize: 604 | | 78 | 6 | Accept | 0.15451 | 2.0399 | 0.153 | 1509 | svm | BoxConstraint: 763.29 | | | | | | | | | | KernelScale: 144.74 | | 79 | 6 | Accept | 0.61916 | 18.894 | 0.153 | 378 | svm | BoxConstraint: 0.043267 | | | | | | | | | | KernelScale: 0.14745 | | 80 | 6 | Accept | 0.17582 | 0.3839 | 0.153 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 81 | 6 | Accept | 0.24677 | 0.93542 | 0.153 | 378 | svm | BoxConstraint: 0.024337 | | | | | | | | | | KernelScale: 76.684 | | 82 | 6 | Accept | 0.17903 | 0.39209 | 0.153 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 83 | 6 | Accept | 0.26744 | 38.332 | 0.153 | 378 | svm | BoxConstraint: 171.89 | | | | | | | | | | KernelScale: 0.0079675 | | 84 | 6 | Accept | 0.20799 | 0.27606 | 0.153 | 378 | tree | MinLeafSize: 20 | | 85 | 6 | Accept | 0.24677 | 12.493 | 0.153 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 225 | | | | | | | | | | MinLeafSize: 8606 | | 86 | 6 | Accept | 0.24677 | 1.0149 | 0.153 | 378 | svm | BoxConstraint: 0.038237 | | | | | | | | | | KernelScale: 63.926 | | 87 | 6 | Accept | 0.24677 | 11.714 | 0.153 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 203 | | | | | | | | | | MinLeafSize: 332 | | 88 | 6 | Best | 0.15053 | 28.585 | 0.15053 | 6033 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 220 | | | | | | | | | | MinLeafSize: 30 | | 89 | 6 | Accept | 0.24677 | 0.19642 | 0.15053 | 378 | tree | MinLeafSize: 683 | | 90 | 6 | Accept | 0.24677 | 14.949 | 0.15053 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 241 | | | | | | | | | | MinLeafSize: 211 | | 91 | 6 | Accept | 0.24677 | 0.97254 | 0.15053 | 378 | svm | BoxConstraint: 0.0035269 | | | | | | | | | | KernelScale: 117.31 | | 92 | 6 | Best | 0.15028 | 13.847 | 0.15028 | 6033 | svm | BoxConstraint: 763.29 | | | | | | | | | | KernelScale: 144.74 | | 93 | 6 | Accept | 0.18829 | 2.5565 | 0.15028 | 1509 | svm | BoxConstraint: 0.0055891 | | | | | | | | | | KernelScale: 0.90898 | | 94 | 6 | Accept | 0.24677 | 12.589 | 0.15028 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 205 | | | | | | | | | | MinLeafSize: 4488 | | 95 | 6 | Accept | 0.19257 | 10.01 | 0.15028 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.92919 | | | | | | | | | | Standardize: false | | 96 | 6 | Accept | 0.1724 | 0.3932 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 97 | 6 | Accept | 0.24677 | 13.185 | 0.15028 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 251 | | | | | | | | | | MinLeafSize: 5425 | | 98 | 6 | Accept | 0.18466 | 0.41239 | 0.15028 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 99 | 6 | Accept | 0.24677 | 0.15762 | 0.15028 | 378 | tree | MinLeafSize: 632 | | 100 | 6 | Accept | 0.18538 | 0.40593 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 101 | 6 | Accept | 0.24735 | 0.1677 | 0.15028 | 378 | tree | MinLeafSize: 159 | | 102 | 6 | Accept | 0.46239 | 19.011 | 0.15028 | 378 | svm | BoxConstraint: 622.29 | | | | | | | | | | KernelScale: 0.13209 | | 103 | 6 | Accept | 0.17613 | 0.43685 | 0.15028 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 104 | 6 | Accept | 0.45428 | 21.612 | 0.15028 | 378 | svm | BoxConstraint: 243.32 | | | | | | | | | | KernelScale: 0.11649 | | 105 | 6 | Accept | 0.17695 | 33.598 | 0.15028 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 21.131 | | | | | | | | | | Standardize: false | | 106 | 6 | Accept | 0.20001 | 0.89752 | 0.15028 | 378 | svm | BoxConstraint: 0.13074 | | | | | | | | | | KernelScale: 4.2955 | | 107 | 6 | Accept | 0.26791 | 34.811 | 0.15028 | 378 | svm | BoxConstraint: 0.15408 | | | | | | | | | | KernelScale: 0.0081934 | | 108 | 6 | Accept | 0.24677 | 12.075 | 0.15028 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 277 | | | | | | | | | | MinLeafSize: 4646 | | 109 | 6 | Accept | 0.17992 | 0.87394 | 0.15028 | 378 | svm | BoxConstraint: 97.764 | | | | | | | | | | KernelScale: 20.653 | | 110 | 6 | Accept | 0.17688 | 0.38115 | 0.15028 | 1509 | tree | MinLeafSize: 42 | | 111 | 6 | Accept | 0.1517 | 13.832 | 0.15028 | 6033 | svm | BoxConstraint: 962.26 | | | | | | | | | | KernelScale: 207.85 | | 112 | 6 | Accept | 0.18863 | 0.4304 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 113 | 6 | Accept | 0.18648 | 0.453 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 114 | 6 | Accept | 0.15876 | 19.737 | 0.15028 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 271 | | | | | | | | | | MinLeafSize: 14 | | 115 | 6 | Accept | 0.24184 | 0.90896 | 0.15028 | 378 | svm | BoxConstraint: 0.26232 | | | | | | | | | | KernelScale: 13.726 | | 116 | 6 | Accept | 0.16007 | 16.574 | 0.15028 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 226 | | | | | | | | | | MinLeafSize: 1 | | 117 | 6 | Accept | 0.18577 | 0.43107 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 118 | 6 | Accept | 0.24677 | 0.2089 | 0.15028 | 378 | tree | MinLeafSize: 112 | | 119 | 6 | Accept | 0.16662 | 23.782 | 0.15028 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 291 | | | | | | | | | | MinLeafSize: 20 | | 120 | 6 | Accept | 0.18481 | 0.3758 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 121 | 6 | Accept | 0.21737 | 13.332 | 0.15028 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 50.518 | | | | | | | | | | Standardize: true | | 122 | 6 | Accept | 0.24677 | 0.92059 | 0.15028 | 378 | svm | BoxConstraint: 0.0042093 | | | | | | | | | | KernelScale: 11.241 | | 123 | 6 | Accept | 0.15316 | 17.743 | 0.15028 | 1509 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 226 | | | | | | | | | | MinLeafSize: 1 | | 124 | 6 | Accept | 0.24677 | 0.16148 | 0.15028 | 378 | tree | MinLeafSize: 302 | | 125 | 6 | Accept | 0.15316 | 21.59 | 0.15028 | 1509 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 271 | | | | | | | | | | MinLeafSize: 14 | | 126 | 6 | Accept | 0.23141 | 37.152 | 0.15028 | 378 | svm | BoxConstraint: 21.777 | | | | | | | | | | KernelScale: 0.029276 | | 127 | 6 | Accept | 0.23304 | 16.815 | 0.15028 | 378 | svm | BoxConstraint: 173.49 | | | | | | | | | | KernelScale: 0.21811 | | 128 | 6 | Accept | 0.24677 | 0.142 | 0.15028 | 378 | tree | MinLeafSize: 2502 | | 129 | 6 | Accept | 0.15706 | 2.1789 | 0.15028 | 1509 | svm | BoxConstraint: 97.764 | | | | | | | | | | KernelScale: 20.653 | | 130 | 6 | Accept | 0.2435 | 12.745 | 0.15028 | 378 | svm | BoxConstraint: 0.053634 | | | | | | | | | | KernelScale: 0.31732 | | 131 | 6 | Accept | 0.21974 | 13.616 | 0.15028 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 3.3625 | | | | | | | | | | Standardize: true | | 132 | 6 | Accept | 0.19986 | 0.42586 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 133 | 6 | Accept | 0.24677 | 0.33075 | 0.15028 | 378 | svm | BoxConstraint: 466.95 | | | | | | | | | | KernelScale: 0.0044945 | | 134 | 6 | Accept | 0.16879 | 0.40206 | 0.15028 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 135 | 6 | Accept | 0.24677 | 0.1607 | 0.15028 | 378 | tree | MinLeafSize: 724 | | 136 | 6 | Accept | 0.15942 | 27.353 | 0.15028 | 1509 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 291 | | | | | | | | | | MinLeafSize: 20 | | 137 | 6 | Accept | 0.1828 | 0.43307 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 138 | 6 | Accept | 0.24677 | 11.762 | 0.15028 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 212 | | | | | | | | | | MinLeafSize: 2353 | | 139 | 6 | Accept | 0.24677 | 14.322 | 0.15028 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 255 | | | | | | | | | | MinLeafSize: 2166 | | 140 | 6 | Accept | 0.19199 | 0.41275 | 0.15028 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 141 | 6 | Accept | 0.18811 | 0.37075 | 0.15028 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 142 | 6 | Accept | 0.24677 | 9.4582 | 0.15028 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 203 | | | | | | | | | | MinLeafSize: 3125 | | 143 | 6 | Accept | 0.24189 | 0.97907 | 0.15028 | 378 | svm | BoxConstraint: 2.2181 | | | | | | | | | | KernelScale: 60.618 | | 144 | 6 | Accept | 0.22596 | 0.81415 | 0.15028 | 378 | svm | BoxConstraint: 8.9199 | | | | | | | | | | KernelScale: 0.93704 | | 145 | 6 | Accept | 0.17702 | 0.43906 | 0.15028 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 146 | 6 | Accept | 0.24749 | 33.011 | 0.15028 | 378 | svm | BoxConstraint: 0.0023424 | | | | | | | | | | KernelScale: 0.02797 | | 147 | 6 | Accept | 0.24677 | 0.3189 | 0.15028 | 378 | svm | BoxConstraint: 0.59521 | | | | | | | | | | KernelScale: 0.0066679 | | 148 | 6 | Accept | 0.24591 | 13.225 | 0.15028 | 378 | svm | BoxConstraint: 521.24 | | | | | | | | | | KernelScale: 0.28175 | | 149 | 6 | Accept | 0.22288 | 0.91802 | 0.15028 | 378 | svm | BoxConstraint: 0.021474 | | | | | | | | | | KernelScale: 4.5183 | | 150 | 6 | Accept | 0.18334 | 0.40628 | 0.15028 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 151 | 6 | Accept | 0.18334 | 13.603 | 0.15028 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 693.96 | | | | | | | | | | Standardize: false | | 152 | 6 | Best | 0.14984 | 34.6 | 0.14984 | 6033 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 271 | | | | | | | | | | MinLeafSize: 14 | | 153 | 6 | Accept | 0.24677 | 0.93891 | 0.14984 | 378 | svm | BoxConstraint: 0.011464 | | | | | | | | | | KernelScale: 887.95 | | 154 | 6 | Accept | 0.24677 | 15.021 | 0.14984 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 271 | | | | | | | | | | MinLeafSize: 3858 | | 155 | 6 | Accept | 0.24677 | 0.2819 | 0.14984 | 378 | svm | BoxConstraint: 0.017 | | | | | | | | | | KernelScale: 0.0046035 | | 156 | 6 | Accept | 0.16761 | 10.948 | 0.14984 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.613 | | | | | | | | | | Standardize: false | | 157 | 6 | Accept | 0.24677 | 0.30553 | 0.14984 | 378 | svm | BoxConstraint: 4.1988 | | | | | | | | | | KernelScale: 0.0011787 | | 158 | 6 | Accept | 0.17316 | 22.007 | 0.14984 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 276 | | | | | | | | | | MinLeafSize: 16 | | 159 | 6 | Accept | 0.20481 | 0.86066 | 0.14984 | 378 | svm | BoxConstraint: 0.01545 | | | | | | | | | | KernelScale: 1.0258 | | 160 | 6 | Accept | 0.15008 | 29.64 | 0.14984 | 6033 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 226 | | | | | | | | | | MinLeafSize: 1 | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 161 | 6 | Accept | 0.18437 | 0.9387 | 0.14984 | 378 | svm | BoxConstraint: 0.0087948 | | | | | | | | | | KernelScale: 1.1494 | | 162 | 6 | Accept | 0.20904 | 13.569 | 0.14984 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 91.024 | | | | | | | | | | Standardize: true | | 163 | 6 | Accept | 0.23451 | 6.5741 | 0.14984 | 378 | svm | BoxConstraint: 0.11495 | | | | | | | | | | KernelScale: 0.34761 | | 164 | 6 | Best | 0.1493 | 147.7 | 0.1493 | 24130 | svm | BoxConstraint: 763.29 | | | | | | | | | | KernelScale: 144.74 | | 165 | 6 | Accept | 0.27656 | 40.771 | 0.1493 | 378 | svm | BoxConstraint: 1.0475 | | | | | | | | | | KernelScale: 0.10172 | | 166 | 6 | Accept | 0.16234 | 26.581 | 0.1493 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.613 | | | | | | | | | | Standardize: false | | 167 | 6 | Accept | 0.19497 | 0.19102 | 0.1493 | 378 | tree | MinLeafSize: 57 | | 168 | 6 | Accept | 0.19417 | 34.123 | 0.1493 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 693.96 | | | | | | | | | | Standardize: false | | 169 | 6 | Accept | 0.25929 | 19.29 | 0.1493 | 378 | svm | BoxConstraint: 19.843 | | | | | | | | | | KernelScale: 0.14367 | | 170 | 6 | Accept | 0.162 | 0.84268 | 0.1493 | 378 | svm | BoxConstraint: 13.978 | | | | | | | | | | KernelScale: 30.295 | | 171 | 6 | Accept | 0.16933 | 0.83709 | 0.1493 | 378 | svm | BoxConstraint: 3.9363 | | | | | | | | | | KernelScale: 17.816 | | 172 | 6 | Accept | 0.15652 | 2.1022 | 0.1493 | 1509 | svm | BoxConstraint: 13.978 | | | | | | | | | | KernelScale: 30.295 | | 173 | 6 | Accept | 0.22197 | 12.696 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 102.13 | | | | | | | | | | Standardize: true | | 174 | 6 | Accept | 0.24677 | 0.18644 | 0.1493 | 378 | tree | MinLeafSize: 864 | | 175 | 6 | Accept | 0.24677 | 0.15003 | 0.1493 | 378 | tree | MinLeafSize: 1322 | | 176 | 6 | Accept | 0.18647 | 8.0636 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.013221 | | | | | | | | | | Standardize: true | | 177 | 6 | Accept | 0.15981 | 2.1353 | 0.1493 | 1509 | svm | BoxConstraint: 3.9363 | | | | | | | | | | KernelScale: 17.816 | | 178 | 6 | Accept | 0.18089 | 0.43329 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 179 | 6 | Accept | 0.17375 | 0.35426 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 180 | 6 | Accept | 0.16252 | 0.38204 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 181 | 6 | Accept | 0.38861 | 33.253 | 0.1493 | 378 | svm | BoxConstraint: 6.5307 | | | | | | | | | | KernelScale: 0.072466 | | 182 | 6 | Accept | 0.17638 | 0.43308 | 0.1493 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 183 | 6 | Accept | 0.15072 | 13.541 | 0.1493 | 6033 | svm | BoxConstraint: 385.35 | | | | | | | | | | KernelScale: 65.814 | | 184 | 6 | Accept | 0.15577 | 25.223 | 0.1493 | 1509 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 276 | | | | | | | | | | MinLeafSize: 16 | | 185 | 6 | Accept | 0.24677 | 14.695 | 0.1493 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 270 | | | | | | | | | | MinLeafSize: 1304 | | 186 | 6 | Accept | 0.19237 | 0.42507 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 187 | 6 | Accept | 0.19325 | 0.38775 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 188 | 6 | Accept | 0.19327 | 9.6642 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.58748 | | | | | | | | | | Standardize: false | | 189 | 6 | Accept | 0.18255 | 13.18 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 54.029 | | | | | | | | | | Standardize: false | | 190 | 6 | Accept | 0.16223 | 0.33709 | 0.1493 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 191 | 6 | Accept | 0.19958 | 0.17893 | 0.1493 | 378 | tree | MinLeafSize: 24 | | 192 | 6 | Accept | 0.24677 | 0.9243 | 0.1493 | 378 | svm | BoxConstraint: 0.028158 | | | | | | | | | | KernelScale: 21.174 | | 193 | 6 | Accept | 0.18379 | 9.5712 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.69661 | | | | | | | | | | Standardize: false | | 194 | 6 | Accept | 0.17841 | 0.41127 | 0.1493 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 195 | 6 | Accept | 0.49303 | 36.252 | 0.1493 | 378 | svm | BoxConstraint: 0.035455 | | | | | | | | | | KernelScale: 0.016789 | | 196 | 6 | Accept | 0.24677 | 0.12499 | 0.1493 | 378 | tree | MinLeafSize: 233 | | 197 | 6 | Accept | 0.24677 | 11.467 | 0.1493 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 209 | | | | | | | | | | MinLeafSize: 398 | | 198 | 6 | Accept | 0.17691 | 0.86872 | 0.1493 | 378 | svm | BoxConstraint: 596.59 | | | | | | | | | | KernelScale: 21.708 | | 199 | 6 | Accept | 0.15712 | 2.9033 | 0.1493 | 1509 | svm | BoxConstraint: 596.59 | | | | | | | | | | KernelScale: 21.708 | | 200 | 6 | Accept | 0.17137 | 0.3622 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 201 | 6 | Accept | 0.24677 | 16.885 | 0.1493 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 299 | | | | | | | | | | MinLeafSize: 216 | | 202 | 6 | Accept | 0.24677 | 11.468 | 0.1493 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 213 | | | | | | | | | | MinLeafSize: 1448 | | 203 | 6 | Accept | 0.16841 | 1.0814 | 0.1493 | 378 | svm | BoxConstraint: 484.15 | | | | | | | | | | KernelScale: 69.555 | | 204 | 5 | Accept | 0.17402 | 13.714 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 3.8082 | | | | | | | | | | Standardize: false | | 205 | 5 | Accept | 0.15642 | 2.2741 | 0.1493 | 1509 | svm | BoxConstraint: 484.15 | | | | | | | | | | KernelScale: 69.555 | | 206 | 6 | Accept | 0.24677 | 0.30604 | 0.1493 | 378 | svm | BoxConstraint: 0.022208 | | | | | | | | | | KernelScale: 0.0071215 | | 207 | 6 | Accept | 0.1817 | 0.42424 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 208 | 6 | Accept | 0.18218 | 0.89991 | 0.1493 | 378 | svm | BoxConstraint: 162.42 | | | | | | | | | | KernelScale: 10.207 | | 209 | 6 | Accept | 0.24677 | 0.92031 | 0.1493 | 378 | svm | BoxConstraint: 28.319 | | | | | | | | | | KernelScale: 317.4 | | 210 | 6 | Accept | 0.17479 | 0.38641 | 0.1493 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 211 | 6 | Accept | 0.225 | 0.86838 | 0.1493 | 378 | svm | BoxConstraint: 810.07 | | | | | | | | | | KernelScale: 3.0301 | | 212 | 6 | Accept | 0.17704 | 0.40973 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 213 | 6 | Accept | 0.19395 | 11.826 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.27286 | | | | | | | | | | Standardize: true | | 214 | 6 | Accept | 0.17788 | 21.7 | 0.1493 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 283 | | | | | | | | | | MinLeafSize: 47 | | 215 | 6 | Accept | 0.14987 | 14.362 | 0.1493 | 6033 | svm | BoxConstraint: 484.15 | | | | | | | | | | KernelScale: 69.555 | | 216 | 6 | Accept | 0.21628 | 13.595 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 9.774 | | | | | | | | | | Standardize: true | | 217 | 5 | Accept | 0.17398 | 0.77326 | 0.1493 | 378 | svm | BoxConstraint: 112.02 | | | | | | | | | | KernelScale: 13.915 | | 218 | 5 | Accept | 0.24677 | 0.1336 | 0.1493 | 378 | tree | MinLeafSize: 653 | | 219 | 6 | Accept | 0.15218 | 40.273 | 0.1493 | 6033 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 276 | | | | | | | | | | MinLeafSize: 16 | | 220 | 6 | Accept | 0.16554 | 0.37094 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 221 | 6 | Accept | 0.15334 | 2.3799 | 0.1493 | 1509 | svm | BoxConstraint: 112.02 | | | | | | | | | | KernelScale: 13.915 | | 222 | 6 | Accept | 0.19759 | 8.0999 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0023918 | | | | | | | | | | Standardize: true | | 223 | 6 | Accept | 0.23132 | 0.77865 | 0.1493 | 378 | svm | BoxConstraint: 10.854 | | | | | | | | | | KernelScale: 0.90505 | | 224 | 6 | Accept | 0.18694 | 8.0651 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.16744 | | | | | | | | | | Standardize: false | | 225 | 6 | Accept | 0.17874 | 0.37035 | 0.1493 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 226 | 6 | Accept | 0.24677 | 0.14836 | 0.1493 | 378 | tree | MinLeafSize: 12624 | | 227 | 6 | Accept | 0.24677 | 11.097 | 0.1493 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 257 | | | | | | | | | | MinLeafSize: 228 | | 228 | 6 | Accept | 0.18473 | 0.45468 | 0.1493 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 229 | 6 | Accept | 0.16683 | 16.222 | 0.1493 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 274 | | | | | | | | | | MinLeafSize: 52 | | 230 | 6 | Accept | 0.24677 | 0.89615 | 0.1493 | 378 | svm | BoxConstraint: 0.0023661 | | | | | | | | | | KernelScale: 236.67 | | 231 | 6 | Accept | 0.21503 | 0.27122 | 0.1493 | 378 | tree | MinLeafSize: 3 | | 232 | 6 | Accept | 0.16733 | 18.627 | 0.1493 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 253 | | | | | | | | | | MinLeafSize: 15 | | 233 | 6 | Accept | 0.16675 | 29.958 | 0.1493 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 3.8082 | | | | | | | | | | Standardize: false | | 234 | 6 | Accept | 0.16016 | 17.053 | 0.1493 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 261 | | | | | | | | | | MinLeafSize: 7 | | 235 | 6 | Accept | 0.16943 | 9.9624 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.4142 | | | | | | | | | | Standardize: false | | 236 | 6 | Accept | 0.24677 | 0.18215 | 0.1493 | 378 | tree | MinLeafSize: 4920 | | 237 | 6 | Accept | 0.24677 | 11.257 | 0.1493 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 268 | | | | | | | | | | MinLeafSize: 2856 | | 238 | 6 | Accept | 0.21822 | 0.1878 | 0.1493 | 378 | tree | MinLeafSize: 62 | | 239 | 6 | Accept | 0.15342 | 21.65 | 0.1493 | 1509 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 274 | | | | | | | | | | MinLeafSize: 52 | | 240 | 6 | Accept | 0.19453 | 8.2977 | 0.1493 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0053849 | | | | | | | | | | Standardize: true | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 241 | 6 | Best | 0.14836 | 96.23 | 0.14836 | 24130 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 271 | | | | | | | | | | MinLeafSize: 14 | | 242 | 6 | Accept | 0.24677 | 0.33161 | 0.14836 | 378 | svm | BoxConstraint: 346.19 | | | | | | | | | | KernelScale: 0.0031468 | | 243 | 6 | Accept | 0.15424 | 20.047 | 0.14836 | 1509 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 261 | | | | | | | | | | MinLeafSize: 7 | | 244 | 6 | Accept | 0.24677 | 0.18571 | 0.14836 | 378 | tree | MinLeafSize: 2558 | | 245 | 6 | Accept | 0.24677 | 0.35537 | 0.14836 | 378 | svm | BoxConstraint: 0.0016007 | | | | | | | | | | KernelScale: 0.0011917 | | 246 | 6 | Accept | 0.17468 | 11.674 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.34418 | | | | | | | | | | Standardize: true | | 247 | 6 | Accept | 0.25459 | 33.254 | 0.14836 | 378 | svm | BoxConstraint: 492.71 | | | | | | | | | | KernelScale: 0.062372 | | 248 | 6 | Accept | 0.24677 | 0.1379 | 0.14836 | 378 | tree | MinLeafSize: 374 | | 249 | 6 | Accept | 0.15195 | 23.675 | 0.14836 | 1509 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 253 | | | | | | | | | | MinLeafSize: 15 | | 250 | 6 | Accept | 0.18005 | 10.469 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.18024 | | | | | | | | | | Standardize: true | | 251 | 6 | Accept | 0.18113 | 10.289 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.15646 | | | | | | | | | | Standardize: true | | 252 | 6 | Accept | 0.14951 | 23.305 | 0.14836 | 6033 | svm | BoxConstraint: 112.02 | | | | | | | | | | KernelScale: 13.915 | | 253 | 6 | Accept | 0.24661 | 0.17732 | 0.14836 | 378 | tree | MinLeafSize: 115 | | 254 | 6 | Accept | 0.24677 | 14.536 | 0.14836 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 275 | | | | | | | | | | MinLeafSize: 7160 | | 255 | 6 | Accept | 0.24677 | 0.14128 | 0.14836 | 378 | tree | MinLeafSize: 342 | | 256 | 6 | Accept | 0.17276 | 0.37834 | 0.14836 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 257 | 6 | Accept | 0.23016 | 12.927 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 35.855 | | | | | | | | | | Standardize: true | | 258 | 6 | Accept | 0.24677 | 0.15532 | 0.14836 | 378 | tree | MinLeafSize: 7345 | | 259 | 6 | Accept | 0.24677 | 0.95577 | 0.14836 | 378 | svm | BoxConstraint: 0.020628 | | | | | | | | | | KernelScale: 164.3 | | 260 | 6 | Accept | 0.2025 | 0.18858 | 0.14836 | 378 | tree | MinLeafSize: 36 | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 261 | 6 | Accept | 0.16554 | 24.966 | 0.14836 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.4142 | | | | | | | | | | Standardize: false | | 262 | 6 | Accept | 0.18069 | 10.339 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.11577 | | | | | | | | | | Standardize: true | | 263 | 6 | Accept | 0.162 | 15.993 | 0.14836 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 245 | | | | | | | | | | MinLeafSize: 2 | | 264 | 6 | Accept | 0.24677 | 11.091 | 0.14836 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 266 | | | | | | | | | | MinLeafSize: 295 | | 265 | 6 | Accept | 0.20847 | 0.27559 | 0.14836 | 378 | tree | MinLeafSize: 2 | | 266 | 6 | Accept | 0.1769 | 30.334 | 0.14836 | 1509 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.34418 | | | | | | | | | | Standardize: true | | 267 | 6 | Accept | 0.24677 | 0.92783 | 0.14836 | 378 | svm | BoxConstraint: 1.8318 | | | | | | | | | | KernelScale: 640.96 | | 268 | 6 | Accept | 0.18559 | 0.38512 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 269 | 6 | Accept | 0.17281 | 17.472 | 0.14836 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 251 | | | | | | | | | | MinLeafSize: 36 | | 270 | 6 | Accept | 0.24677 | 0.13322 | 0.14836 | 378 | tree | MinLeafSize: 1065 | | 271 | 6 | Accept | 0.23467 | 33.997 | 0.14836 | 378 | svm | BoxConstraint: 40.942 | | | | | | | | | | KernelScale: 0.060508 | | 272 | 6 | Accept | 0.16424 | 0.40692 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 273 | 6 | Accept | 0.16911 | 25.586 | 0.14836 | 1509 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 283 | | | | | | | | | | MinLeafSize: 47 | | 274 | 6 | Accept | 0.17906 | 0.39597 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 275 | 6 | Accept | 0.18946 | 8.6561 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0097303 | | | | | | | | | | Standardize: true | | 276 | 6 | Accept | 0.17244 | 0.39582 | 0.14836 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 277 | 6 | Accept | 0.19106 | 0.17636 | 0.14836 | 378 | tree | MinLeafSize: 58 | | 278 | 6 | Accept | 0.15421 | 18.602 | 0.14836 | 1509 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 245 | | | | | | | | | | MinLeafSize: 2 | | 279 | 6 | Accept | 0.1515 | 34.698 | 0.14836 | 6033 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 253 | | | | | | | | | | MinLeafSize: 15 | | 280 | 6 | Accept | 0.18897 | 13.381 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 697.61 | | | | | | | | | | Standardize: true | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 281 | 6 | Accept | 0.16858 | 0.37927 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 282 | 6 | Accept | 0.22522 | 13.462 | 0.14836 | 378 | svm | BoxConstraint: 0.07043 | | | | | | | | | | KernelScale: 0.25792 | | 283 | 6 | Accept | 0.2417 | 1.0039 | 0.14836 | 378 | svm | BoxConstraint: 0.0013375 | | | | | | | | | | KernelScale: 3.417 | | 284 | 6 | Accept | 0.18405 | 0.3932 | 0.14836 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 285 | 6 | Accept | 0.24677 | 0.33159 | 0.14836 | 378 | svm | BoxConstraint: 0.038245 | | | | | | | | | | KernelScale: 0.0021108 | | 286 | 6 | Accept | 0.19548 | 0.43701 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 287 | 6 | Accept | 0.24677 | 1.0456 | 0.14836 | 378 | svm | BoxConstraint: 0.0050787 | | | | | | | | | | KernelScale: 664.76 | | 288 | 6 | Accept | 0.15713 | 0.41854 | 0.14836 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 289 | 6 | Accept | 0.24677 | 12.752 | 0.14836 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 232 | | | | | | | | | | MinLeafSize: 3957 | | 290 | 6 | Accept | 0.17541 | 0.38528 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 291 | 6 | Accept | 0.16182 | 23.699 | 0.14836 | 1509 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 251 | | | | | | | | | | MinLeafSize: 36 | | 292 | 6 | Accept | 0.20209 | 8.1049 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.036333 | | | | | | | | | | Standardize: false | | 293 | 6 | Accept | 0.24677 | 9.4156 | 0.14836 | 378 | ensemble | Method: LogitBoost | | | | | | | | | | NumLearningCycles: 208 | | | | | | | | | | MinLeafSize: 2039 | | 294 | 6 | Accept | 0.1795 | 0.41664 | 0.14836 | 1509 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 295 | 5 | Accept | 0.20244 | 8.2445 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.003178 | | | | | | | | | | Standardize: false | | 296 | 5 | Accept | 0.22113 | 0.91625 | 0.14836 | 378 | svm | BoxConstraint: 83.258 | | | | | | | | | | KernelScale: 180.57 | | 297 | 6 | Accept | 0.24677 | 0.15149 | 0.14836 | 378 | tree | MinLeafSize: 330 | | 298 | 6 | Accept | 0.20546 | 0.23764 | 0.14836 | 378 | tree | MinLeafSize: 3 | | 299 | 6 | Accept | 0.17061 | 0.40961 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | | 300 | 6 | Accept | 0.19405 | 0.41665 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | | | Standardize: - | |====================================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Training set | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | size | | | |====================================================================================================================================================| | 301 | 6 | Accept | 0.16126 | 17.363 | 0.14836 | 378 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 216 | | | | | | | | | | MinLeafSize: 2 | | 302 | 6 | Accept | 0.24677 | 0.1915 | 0.14836 | 378 | tree | MinLeafSize: 572 | | 303 | 6 | Accept | 0.20638 | 12.701 | 0.14836 | 378 | nb | DistributionNames: kernel | | | | | | | | | | Width: 4.8773 | | | | | | | | | | Standardize: false | | 304 | 6 | Accept | 0.15775 | 0.43674 | 0.14836 | 378 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | | | | | | | ...
__________________________________________________________ Optimization completed. Total iterations: 425 Total elapsed time: 693.4747 seconds Total time for training and validation: 3784.6551 seconds Best observed learner is an ensemble model with: Learner: ensemble Method: LogitBoost NumLearningCycles: 271 MinLeafSize: 14 Observed validation loss: 0.14836 Time for training and validation: 96.2296 seconds Documentation for fitcauto display
The Total elapsed time value shows that the ASHA optimization took less time to run than the Bayesian optimization (about 0.2 hours).
The final model returned by fitcauto corresponds to the best observed learner. Before returning the model, the function retrains it using the entire training data set (adultdata), the listed Learner (or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
Evaluate the performance of the returned bayesianMdl and ashaMdl models on the test set adulttest by using confusion matrices and receiver operating characteristic (ROC) curves.
For each model, find the predicted labels and score values for the test set.
[bayesianLabels,bayesianScores] = predict(bayesianMdl,adulttest); [ashaLabels,ashaScores] = predict(ashaMdl,adulttest);
Create confusion matrices from the test set results. The diagonal elements indicate the number of correctly classified instances of a given class. The off-diagonal elements are instances of misclassified observations. Use a 1-by-2 tiled layout to compare the results.
tiledlayout(1,2) nexttile confusionchart(adulttest.salary,bayesianLabels) title("Bayesian Optimization") nexttile confusionchart(adulttest.salary,ashaLabels) title("ASHA Optimization")
Compute the test set classification accuracy for each model, where the accuracy is the percentage of correctly classified test set observations.
bayesianAccuracy = (1-loss(bayesianMdl,adulttest,"salary"))*100bayesianAccuracy = 85.3243
ashaAccuracy = (1-loss(ashaMdl,adulttest,"salary"))*100ashaAccuracy = 85.4724
Based on the confusion matrices and the accuracy values, both models perform well.
For each model, plot the ROC curve and compute the area under the ROC curve (AUC). The ROC curve shows the true positive rate versus the false positive rate for different thresholds of classification scores. For a perfect classifier, whose true positive rate is always 1 regardless of the threshold, AUC = 1. For a binary classifier that randomly assigns observations to classes, AUC = 0.5. A large AUC value (close to 1) indicates good classifier performance.
For each model, compute the metrics for the ROC curve and find the AUC value by creating a rocmetrics object.
bayesianROC = rocmetrics(adulttest.salary,bayesianScores,bayesianMdl.ClassNames); ashaROC = rocmetrics(adulttest.salary,ashaScores,ashaMdl.ClassNames);
Plot the ROC curves for the label <=50K by using the plot function of rocmetrics.
figure [r1,g1] = plot(bayesianROC,"ClassNames","<=50K"); hold on [r2,g2] = plot(ashaROC,"ClassNames","<=50K"); r1.DisplayName = replace(r1.DisplayName,"<=50K","Bayesian Optimization"); r2.DisplayName = replace(r2.DisplayName,"<=50K","ASHA Optimization"); g1(1).DisplayName = "Bayesian Optimization Model Operating Point"; g2(1).DisplayName = "ASHA Optimization Model Operating Point"; title("ROC Curves for Class <=50K") hold off
Based on the AUC values, both classifiers perform well on the test data.
See Also
fitcauto | confusionchart | perfcurve | BayesianOptimization
