Automated Regression Model Selection with Bayesian Optimization

This example shows how to use the `fitrauto` function to automatically try a selection of regression model types with different hyperparameter values, given training predictor and response data. The function uses Bayesian optimization to select models and their hyperparameter values, and computes the following for each model: $\mathrm{log}\left(1+valLoss\right)$, where valLoss is the cross-validation mean squared error (MSE). After the optimization is complete, `fitrauto` returns the model, trained on the entire data set, that is expected to best predict the responses for new data. Check the model performance on test data.

Prepare Data

Load the sample data set `NYCHousing2015`, which includes 10 variables with information on the sales of properties in New York City in 2015. This example uses some of these variables to analyze the sale prices.

`load NYCHousing2015`

Instead of loading the sample data set `NYCHousing2015`, you can download the data from the NYC Open Data website and import the data as follows.

```folder = 'Annualized_Rolling_Sales_Update'; ds = spreadsheetDatastore(folder,"TextType","string","NumHeaderLines",4); ds.Files = ds.Files(contains(ds.Files,"2015")); ds.SelectedVariableNames = ["BOROUGH","NEIGHBORHOOD","BUILDINGCLASSCATEGORY","RESIDENTIALUNITS", ... "COMMERCIALUNITS","LANDSQUAREFEET","GROSSSQUAREFEET","YEARBUILT","SALEPRICE","SALEDATE"]; NYCHousing2015 = readall(ds); ```

Preprocess the data set to choose the predictor variables of interest. Some of the preprocessing steps match those in the example Train Linear Regression Model.

First, change the variable names to lowercase for readability.

`NYCHousing2015.Properties.VariableNames = lower(NYCHousing2015.Properties.VariableNames);`

Next, remove samples with certain problematic values. For example, retain only those samples where at least one of the area measurements `grosssquarefeet` or `landsquarefeet` is nonzero. Assume that a `saleprice` of \$0 indicates an ownership transfer without a cash consideration, and remove the samples with that `saleprice` value. Assume that a `yearbuilt` value of 1500 or less is a typo, and remove the corresponding samples.

```NYCHousing2015(NYCHousing2015.grosssquarefeet == 0 & NYCHousing2015.landsquarefeet == 0,:) = []; NYCHousing2015(NYCHousing2015.saleprice == 0,:) = []; NYCHousing2015(NYCHousing2015.yearbuilt <= 1500,:) = [];```

Convert the `saledate` variable, specified as a `datetime` array, into two numeric columns `MM` (month) and `DD` (day), and remove the `saledate` variable. Ignore the year values because all samples are for the year 2015.

```[~,NYCHousing2015.MM,NYCHousing2015.DD] = ymd(NYCHousing2015.saledate); NYCHousing2015.saledate = [];```

The numeric values in the `borough` variable indicate the names of the boroughs. Change the variable to a categorical variable using the names.

```NYCHousing2015.borough = categorical(NYCHousing2015.borough,1:5, ... ["Manhattan","Bronx","Brooklyn","Queens","Staten Island"]);```

The `neighborhood` variable has 254 categories. Remove this variable for simplicity.

`NYCHousing2015.neighborhood = [];`

Convert the `buildingclasscategory` variable to a categorical variable, and explore the variable by using the `wordcloud` function.

```NYCHousing2015.buildingclasscategory = categorical(NYCHousing2015.buildingclasscategory); wordcloud(NYCHousing2015.buildingclasscategory);```

Assume that you are interested only in one-, two-, and three-family dwellings. Find the sample indices for these dwellings and delete the other samples. Then, change the `buildingclasscategory` variable to an ordinal categorical variable, with integer-valued category names.

```idx = ismember(string(NYCHousing2015.buildingclasscategory), ... ["01 ONE FAMILY DWELLINGS","02 TWO FAMILY DWELLINGS","03 THREE FAMILY DWELLINGS"]); NYCHousing2015 = NYCHousing2015(idx,:); NYCHousing2015.buildingclasscategory = categorical(NYCHousing2015.buildingclasscategory, ... ["01 ONE FAMILY DWELLINGS","02 TWO FAMILY DWELLINGS","03 THREE FAMILY DWELLINGS"], ... ["1","2","3"],'Ordinal',true);```

The `buildingclasscategory` variable now indicates the number of families in one dwelling.

Explore the response variable `saleprice` by using the `summary` function.

```s = summary(NYCHousing2015); s.saleprice```
```ans = struct with fields: Size: [24972 1] Type: 'double' Description: '' Units: '' Continuity: [] Min: 1 Median: 515000 Max: 37000000 NumMissing: 0 ```

Create a histogram of the `saleprice` variable.

`histogram(NYCHousing2015.saleprice)`

Because the distribution of `saleprice` values is right-skewed, with all values greater than 0, log transform the `saleprice` variable.

`NYCHousing2015.saleprice = log(NYCHousing2015.saleprice);`

Similarly, transform the `grosssquarefeet` and `landsquarefeet` variables. Add a value of 1 before taking the logarithm of each variable, in case the variable is equal to 0.

```NYCHousing2015.grosssquarefeet = log(1 + NYCHousing2015.grosssquarefeet); NYCHousing2015.landsquarefeet = log(1 + NYCHousing2015.landsquarefeet);```

Partition Data and Remove Outliers

Partition the data set into a training set and a test set by using `cvpartition`. Use approximately 80% of the observations for the model selection and hyperparameter tuning process, and the other 20% to test the performance of the final model returned by `fitrauto`.

```rng('default') % For reproducibility of the partition c = cvpartition(length(NYCHousing2015.saleprice),'Holdout',0.2); trainData = NYCHousing2015(training(c),:); testData = NYCHousing2015(test(c),:);```

Identify and remove the outliers of `saleprice`, `grosssquarefeet`, and `landsquarefeet` from the training data by using the `isoutlier` function.

```[priceIdx,priceL,priceU] = isoutlier(trainData.saleprice); trainData(priceIdx,:) = []; [grossIdx,grossL,grossU] = isoutlier(trainData.grosssquarefeet); trainData(grossIdx,:) = []; [landIdx,landL,landU] = isoutlier(trainData.landsquarefeet); trainData(landIdx,:) = [];```

Remove the outliers of `saleprice`, `grosssquarefeet`, and `landsquarefeet` from the test data by using the same lower and upper thresholds computed on the training data.

```testData(testData.saleprice < priceL | testData.saleprice > priceU,:) = []; testData(testData.grosssquarefeet < grossL | testData.grosssquarefeet > grossU,:) = []; testData(testData.landsquarefeet < landL | testData.landsquarefeet > landU,:) = [];```

Use Automated Model Selection

Find an appropriate regression model for the data in `trainData` by using `fitrauto`. Try tree and ensemble learners and run the Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. To reduce computational time, use 3-fold cross-validation, rather than 5-fold cross-validation, as part of the optimization process.

Because of the complexity of the optimization, this process can take some time, especially for larger data sets. By default, `fitrauto` provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.

```options = struct('UseParallel',true,'Kfold',3); [mdl,results] = fitrauto(trainData,'saleprice', ... 'Learners',{'tree','ensemble'},'HyperparameterOptimizationOptions',options);```
```Copying objective function to workers... Done copying objective function to workers. ```
```Learner types to explore: ensemble, tree Total iterations (MaxObjectiveEvaluations): 60 Total time (MaxTime): Inf ```
```|===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 1 | 5 | Accept | 0.25922 | 0.067333 | 0.18985 | 0.19575 | tree | MinLeafSize: 8676 | | 2 | 5 | Best | 0.18985 | 0.14568 | 0.18985 | 0.19575 | tree | MinLeafSize: 245 | ```
```| 3 | 2 | Accept | 0.25126 | 0.86908 | 0.1849 | 0.18985 | tree | MinLeafSize: 4 | | 4 | 2 | Best | 0.1849 | 1.0049 | 0.1849 | 0.18985 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 18 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 110 | | 5 | 2 | Accept | 0.25922 | 0.5705 | 0.1849 | 0.18985 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 24 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 8426 | | 6 | 2 | Accept | 0.25126 | 0.87986 | 0.1849 | 0.18985 | tree | MinLeafSize: 4 | ```
```| 7 | 6 | Accept | 0.21227 | 0.069611 | 0.1849 | 0.18985 | tree | MinLeafSize: 1722 | ```
```| 8 | 4 | Accept | 0.18763 | 0.15728 | 0.1849 | 0.18538 | tree | MinLeafSize: 60 | | 9 | 4 | Accept | 2.9803 | 0.47009 | 0.1849 | 0.18538 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 20 | | | | | | | | | | LearnRate: 0.054589 | | | | | | | | | | MinLeafSize: 8499 | | 10 | 4 | Accept | 0.1914 | 0.23832 | 0.1849 | 0.18538 | tree | MinLeafSize: 30 | ```
```|===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 11 | 4 | Accept | 0.23248 | 0.057309 | 0.1849 | 0.18494 | tree | MinLeafSize: 3991 | ```
```| 12 | 4 | Accept | 0.22354 | 2.9816 | 0.1849 | 0.18493 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 121 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 3071 | ```
```| 13 | 4 | Accept | 4.7611 | 3.9969 | 0.1849 | 0.18494 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 79 | | | | | | | | | | LearnRate: 0.0025753 | | | | | | | | | | MinLeafSize: 67 | ```
```| 14 | 3 | Best | 0.17937 | 1.6426 | 0.17937 | 0.17945 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 24 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 16 | | 15 | 3 | Accept | 0.25923 | 0.31222 | 0.17937 | 0.17945 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 14 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 7287 | ```
```| 16 | 5 | Best | 0.17799 | 4.7324 | 0.17799 | 0.17945 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 69 | | | | | | | | | | LearnRate: 0.19523 | | | | | | | | | | MinLeafSize: 209 | | 17 | 5 | Accept | 0.19076 | 0.10145 | 0.17799 | 0.17945 | tree | MinLeafSize: 274 | ```
```| 18 | 4 | Accept | 0.22517 | 0.079648 | 0.17799 | 0.17945 | tree | MinLeafSize: 2306 | | 19 | 4 | Accept | 0.21507 | 0.32105 | 0.17799 | 0.17945 | tree | MinLeafSize: 10 | ```
```| 20 | 4 | Accept | 0.18797 | 0.11404 | 0.17799 | 0.17945 | tree | MinLeafSize: 155 | ```
```|===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 21 | 3 | Accept | 0.17862 | 5.6403 | 0.17799 | 0.17945 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 54 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 3 | | 22 | 3 | Accept | 0.19413 | 0.16212 | 0.17799 | 0.17945 | tree | MinLeafSize: 25 | ```
```| 23 | 6 | Accept | 0.24396 | 0.74378 | 0.17799 | 0.17945 | tree | MinLeafSize: 5 | ```
```| 24 | 5 | Accept | 0.18986 | 0.16919 | 0.17799 | 0.17945 | tree | MinLeafSize: 39 | | 25 | 5 | Accept | 0.19608 | 0.19077 | 0.17799 | 0.17945 | tree | MinLeafSize: 23 | ```
```| 26 | 4 | Accept | 0.17828 | 11.436 | 0.17799 | 0.17803 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 108 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 3 | | 27 | 4 | Accept | 0.1809 | 3.7272 | 0.17799 | 0.17803 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 69 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 55 | ```
```| 28 | 4 | Accept | 0.18171 | 1.9361 | 0.17799 | 0.17803 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 19 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 4 | ```
```| 29 | 4 | Accept | 0.17959 | 8.6553 | 0.17799 | 0.17803 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 75 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 1 | ```
```| 30 | 3 | Accept | 0.20204 | 15.893 | 0.17762 | 0.17676 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 496 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 910 | |===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 31 | 3 | Best | 0.17762 | 6.6202 | 0.17762 | 0.17676 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 95 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 13 | ```
```| 32 | 5 | Accept | 0.19444 | 5.563 | 0.17762 | 0.17765 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 103 | | | | | | | | | | LearnRate: 0.9936 | | | | | | | | | | MinLeafSize: 98 | | 33 | 5 | Accept | 0.18056 | 0.75592 | 0.17762 | 0.17765 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 11 | | | | | | | | | | LearnRate: 0.49541 | | | | | | | | | | MinLeafSize: 222 | ```
```| 34 | 4 | Accept | 0.18768 | 0.78702 | 0.17762 | 0.17765 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 13 | | | | | | | | | | LearnRate: 0.98545 | | | | | | | | | | MinLeafSize: 1 | | 35 | 4 | Accept | 0.26635 | 1.1019 | 0.17762 | 0.17765 | tree | MinLeafSize: 2 | ```
```| 36 | 4 | Accept | 0.206 | 4.2801 | 0.17762 | 0.17783 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 142 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 1241 | ```
```| 37 | 3 | Accept | 0.21503 | 11.309 | 0.17762 | 0.17764 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 230 | | | | | | | | | | LearnRate: 0.017904 | | | | | | | | | | MinLeafSize: 1 | | 38 | 3 | Accept | 0.3789 | 0.55978 | 0.17762 | 0.17764 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 10 | | | | | | | | | | LearnRate: 0.27761 | | | | | | | | | | MinLeafSize: 5 | ```
```| 39 | 6 | Accept | 0.23053 | 0.51383 | 0.17762 | 0.17764 | tree | MinLeafSize: 7 | ```
```| 40 | 6 | Accept | 0.17996 | 2.7932 | 0.17762 | 0.17764 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 49 | | | | | | | | | | LearnRate: 0.11514 | | | | | | | | | | MinLeafSize: 1 | ```
```|===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 41 | 6 | Accept | 0.23707 | 12.289 | 0.17762 | 0.17767 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 480 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 3779 | ```
```| 42 | 5 | Accept | 0.18527 | 26.878 | 0.17762 | 0.17766 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 491 | | | | | | | | | | LearnRate: 0.65315 | | | | | | | | | | MinLeafSize: 1044 | | 43 | 5 | Accept | 0.18276 | 4.678 | 0.17762 | 0.17766 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 66 | | | | | | | | | | LearnRate: 0.81673 | | | | | | | | | | MinLeafSize: 635 | ```
```| 44 | 5 | Accept | 0.25057 | 22.826 | 0.17762 | 0.17765 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 412 | | | | | | | | | | LearnRate: 0.92469 | | | | | | | | | | MinLeafSize: 1 | ```
```| 45 | 5 | Accept | 4.61 | 0.60326 | 0.17762 | 0.17779 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 10 | | | | | | | | | | LearnRate: 0.027624 | | | | | | | | | | MinLeafSize: 1 | ```
```| 46 | 4 | Accept | 0.2001 | 25.14 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 465 | | | | | | | | | | LearnRate: 0.0094768 | | | | | | | | | | MinLeafSize: 23 | | 47 | 4 | Accept | 0.20319 | 3.6011 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 70 | | | | | | | | | | LearnRate: 0.98445 | | | | | | | | | | MinLeafSize: 4 | ```
```| 48 | 4 | Accept | 0.18106 | 26.642 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 495 | | | | | | | | | | LearnRate: 0.091101 | | | | | | | | | | MinLeafSize: 1 | ```
```| 49 | 4 | Accept | 0.25922 | 7.6886 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 493 | | | | | | | | | | LearnRate: 0.024602 | | | | | | | | | | MinLeafSize: 8658 | ```
```| 50 | 4 | Accept | 0.18162 | 4.8133 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 94 | | | | | | | | | | LearnRate: 0.30945 | | | | | | | | | | MinLeafSize: 1 | ```
```|===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 51 | 5 | Accept | 0.17788 | 5.18 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 102 | | | | | | | | | | LearnRate: 0.11411 | | | | | | | | | | MinLeafSize: 4 | ```
```| 52 | 5 | Accept | 0.1858 | 36.145 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 499 | | | | | | | | | | LearnRate: 0.34801 | | | | | | | | | | MinLeafSize: 181 | | 53 | 5 | Accept | 0.17946 | 1.3924 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 24 | | | | | | | | | | LearnRate: 0.47121 | | | | | | | | | | MinLeafSize: 29 | ```
```| 54 | 6 | Accept | 0.17837 | 8.0619 | 0.17762 | 0.17761 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 142 | | | | | | | | | | LearnRate: 0.076159 | | | | | | | | | | MinLeafSize: 1 | ```
```| 55 | 6 | Accept | 0.17766 | 25.949 | 0.17762 | 0.17761 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 495 | | | | | | | | | | LearnRate: 0.024193 | | | | | | | | | | MinLeafSize: 7 | ```
```| 56 | 6 | Accept | 1.2951 | 25.877 | 0.17762 | 0.17762 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 475 | | | | | | | | | | LearnRate: 0.0044769 | | | | | | | | | | MinLeafSize: 5 | ```
```| 57 | 6 | Best | 0.17753 | 28.063 | 0.17753 | 0.17751 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 486 | | | | | | | | | | LearnRate: 0.038349 | | | | | | | | | | MinLeafSize: 39 | ```
```| 58 | 6 | Accept | 0.17978 | 2.0228 | 0.17753 | 0.17751 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 34 | | | | | | | | | | LearnRate: 0.3482 | | | | | | | | | | MinLeafSize: 3 | ```
```| 59 | 6 | Accept | 2.6442 | 0.64641 | 0.17753 | 0.17751 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 10 | | | | | | | | | | LearnRate: 0.12206 | | | | | | | | | | MinLeafSize: 65 | ```
```| 60 | 5 | Accept | 0.97054 | 16.16 | 0.17753 | 0.17751 | ensemble | Method: LSBoost | | | | | | | | | | NumLearningCycles: 498 | | | | | | | | | | LearnRate: 0.0048163 | | | | | | | | | | MinLeafSize: 3269 | |===================================================================================================================================================| | Iter | Active | Eval | log(1 + valLoss) | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | | & validation (sec)| log(1 + valLoss) | log(1 + valLoss) | | | |===================================================================================================================================================| | 61 | 5 | Accept | 0.2084 | 0.6174 | 0.17753 | 0.17751 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 15 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 997 | ```

```__________________________________________________________ Optimization completed. Total iterations: 61 Total elapsed time: 150.2458 seconds Total time for training and validation: 386.9239 seconds Best observed learner is an ensemble model with: Method: LSBoost NumLearningCycles: 486 LearnRate: 0.038349 MinLeafSize: 39 Observed log(1 + valLoss): 0.17753 Time for training and validation: 28.0634 seconds Best estimated learner (returned model) is an ensemble model with: Method: LSBoost NumLearningCycles: 486 LearnRate: 0.038349 MinLeafSize: 39 Estimated log(1 + valLoss): 0.17751 Estimated time for training and validation: 28.7325 seconds Documentation for fitrauto display ```

The final model returned by `fitrauto` corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (`trainData`), the listed `Learner` (or model) type, and the displayed hyperparameter values.

Evaluate Test Set Performance

Evaluate the performance of the returned model `mdl` on the test set `testData`. Compute the test set mean squared error (MSE), and take a log transform of the MSE to match the values in the verbose display of `fitrauto`. Smaller MSE (and log-transformed MSE) values indicate better performance.

```testMSE = loss(mdl,testData,'saleprice'); testError = log(1 + testMSE)```
```testError = 0.1791 ```

Compare the predicted test set response values to the true response values. Plot the predicted sale price along the vertical axis and the true sale price along the horizontal axis. Points on the reference line indicate correct predictions. A good model produces predictions that are scattered near the line.

```testPredictions = predict(mdl,testData); plot(testData.saleprice,testPredictions,'.') hold on plot(testData.saleprice,testData.saleprice) % Reference line hold off xlabel(["True Sale Price","(log transformed)"]) ylabel(["Predicted Sale Price","(log transformed)"])```

Use box plots to compare the distribution of predicted and true sale prices by borough. Create the box plots by using the `boxchart` function. Each box plot displays the median, the lower and upper quartiles, any outliers (computed using the interquartile range), and the minimum and maximum values that are not outliers. In particular, the line inside each box is the sample median, and the circular markers indicate outliers.

For each borough, compare the red box plot (showing the distribution of predicted prices) to the blue box plot (showing the distribution of true prices). Similar distributions for the predicted and true sale prices indicate good predictions.

```boxchart(testData.borough,testData.saleprice) hold on boxchart(testData.borough,testPredictions) hold off legend(["True Sale Prices","Predicted Sale Prices"]) xlabel("Borough") ylabel(["Sale Price","(log transformed)"])```

For all the boroughs, the predicted median sale price closely matches the median true sale price. The predicted sale prices seem to vary less than the true sale prices.

Display box charts that compare the distribution of predicted and true sale prices by the number of families in a dwelling.

```boxchart(testData.buildingclasscategory,testData.saleprice) hold on boxchart(testData.buildingclasscategory,testPredictions) hold off legend(["True Sale Prices","Predicted Sale Prices"]) xlabel("Number of Families in Dwelling") ylabel(["Sale Price","(log transformed)"])```

For all dwellings, the predicted median sale price closely matches the median true sale price. The predicted sale prices seem to vary less than the true sale prices.

Plot a histogram of the test set residuals, and check that they are normally distributed. (Recall that the sale prices are log-transformed.)

```testResiduals = testData.saleprice - testPredictions; histogram(testResiduals) title('Test Set Residuals')```

Although the histogram is slightly left-skewed, it is approximately symmetric about 0.