kfoldMargin

Classification margins for cross-validated classification model

Syntax

M = kfoldMargin(CVMdl)

M = kfoldMargin(CVMdl,'IncludeInteractions',includeInteractions)

Description

M = kfoldMargin(CVMdl) returns classification margins obtained by the cross-validated classification model CVMdl. For every fold, kfoldMargin computes classification margins for validation-fold observations using a classifier trained on training-fold observations. CVMdl.X and CVMdl.Y contain both sets of observations.

M = kfoldMargin(CVMdl,'IncludeInteractions',includeInteractions) specifies whether to include interaction terms in computations. This syntax applies only to generalized additive models.

Examples

Estimate k-fold Margins of Classifier

Find the k-fold margins for an ensemble that classifies the ionosphere data.

Load the ionosphere data set.

load ionosphere

Create a template tree stump.

t = templateTree('MaxNumSplits',1);

Train a classification ensemble of decision trees. Specify t as the weak learner.

Mdl = fitcensemble(X,Y,'Method','AdaBoostM1','Learners',t);

Cross-validate the classifier using 10-fold cross-validation.

cvens = crossval(Mdl);

Compute the k-fold margins. Display summary statistics for the margins.

m = kfoldMargin(cvens);
marginStats = table(min(m),mean(m),max(m),...
    'VariableNames',{'Min','Mean','Max'})

marginStats=1×3 table
      Min       Mean      Max  
    _______    ______    ______

    -11.312    7.3236    23.517

Input Arguments

CVMdl — Cross-validated partitioned classifier

ClassificationPartitionedModel object | ClassificationPartitionedEnsemble object | ClassificationPartitionedGAM object

Cross-validated partitioned classifier, specified as a ClassificationPartitionedModel, ClassificationPartitionedEnsemble, or ClassificationPartitionedGAM object. You can create the object in two ways:

• Pass a trained classification model listed in the following table to its crossval object function.

• Train a classification model using a function listed in the following table and specify one of the cross-validation name-value arguments for the function.

Classification Model	Function
ClassificationDiscriminant	fitcdiscr
ClassificationEnsemble	fitcensemble
ClassificationGAM	fitcgam
ClassificationKNN	fitcknn
ClassificationNaiveBayes	fitcnb
ClassificationNeuralNetwork	fitcnet
ClassificationSVM	fitcsvm
ClassificationTree	fitctree

includeInteractions — Flag to include interaction terms

true | false

Flag to include interaction terms of the model, specified as true or false. This argument is valid only for a generalized additive model (GAM). That is, you can specify this argument only when CVMdl is ClassificationPartitionedGAM.

The default value is true if the models in CVMdl (CVMdl.Trained) contain interaction terms. The value must be false if the models do not contain interaction terms.

Data Types: logical

Output Arguments

M — Classification margins

numeric vector

Classification margins, returned as a numeric vector. M is an n-by-1 vector, where each row is the margin of the corresponding observation and n is the number of observations. (n is size(CVMdl.X,1) when observations are in rows.)

If you use a holdout validation technique to create CVMdl (that is, if CVMdl.KFold is 1), then M has NaN values for training-fold observations.

More About

Classification Margin

The classification margin for binary classification is, for each observation, the difference between the classification score for the true class and the classification score for the false class. The classification margin for multiclass classification is the difference between the classification score for the true class and the maximal score for the false classes.

If the margins are on the same scale (that is, the score values are based on the same score transformation), then they serve as a classification confidence measure. Among multiple classifiers, those that yield greater margins are better.

Algorithms

kfoldMargin computes classification margins as described in the corresponding margin object function. For a model-specific description, see the appropriate margin function reference page in the following table.

Model Type	margin Function
Discriminant analysis classifier	margin
Ensemble classifier	margin
Generalized additive model classifier	margin
k-nearest neighbor classifier	margin
Naive Bayes classifier	margin
Neural network classifier	margin
Support vector machine classifier	margin
Binary decision tree for multiclass classification	margin

Extended Capabilities

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

Usage notes and limitations:

• This function fully supports GPU arrays for the following cross-validated model objects:

  • Ensemble classifier trained with fitcensemble

  • k-nearest neighbor classifier trained with fitcknn

  • Support vector machine classifier trained with fitcsvm

  • Binary decision tree for multiclass classification trained with fitctree

  • Neural network for classification trained with fitcnet

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2011a

R2024b: Specify GPU arrays for neural network models (requires Parallel Computing Toolbox)

kfoldMargin fully supports GPU arrays for ClassificationPartitionedModel models trained using fitcnet.

R2023b: Observations with missing predictor values are used in resubstitution and cross-validation computations

Starting in R2023b, the following classification model object functions use observations with missing predictor values as part of resubstitution ("resub") and cross-validation ("kfold") computations for classification edges, losses, margins, and predictions.

Model Type	Model Objects	Object Functions
Discriminant analysis classification model	ClassificationDiscriminant	resubEdge, resubLoss, resubMargin, resubPredict
	ClassificationPartitionedModel	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
Ensemble of discriminant analysis learners for classification	ClassificationEnsemble	resubEdge, resubLoss, resubMargin, resubPredict
	ClassificationPartitionedEnsemble	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
Gaussian kernel classification model	ClassificationPartitionedKernel	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
	ClassificationPartitionedKernelECOC	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
Linear classification model	ClassificationPartitionedLinear	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
	ClassificationPartitionedLinearECOC	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
Neural network classification model	ClassificationNeuralNetwork	resubEdge, resubLoss, resubMargin, resubPredict
	ClassificationPartitionedModel	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict
Support vector machine (SVM) classification model	ClassificationSVM	resubEdge, resubLoss, resubMargin, resubPredict
	ClassificationPartitionedModel	kfoldEdge, kfoldLoss, kfoldMargin, kfoldPredict

In previous releases, the software omitted observations with missing predictor values from the resubstitution and cross-validation computations.