risk.validation.brierScore

Return Brier score for probability data

Since R2025a

Syntax

brierValue = risk.validation.brierScore(Probability,BinaryResponse)

[brierValue,Output] = risk.validation.brierScore(Probability,BinaryResponse)

Description

brierValue = risk.validation.brierScore(Probability,BinaryResponse) returns the Brier score, where Probability contains numeric values that represent quantities such as probability of default (PD) estimates. BinaryResponse specifies the target state of each value in Probability.

[brierValue,Output] = risk.validation.brierScore(Probability,BinaryResponse) returns a structure with the fields BrierScore, MeanResponse, and ReferenceScore.

Examples

Compute Brier Score for Probability of Default

Compute the Brier score for a set of probability of default (PD) values by using the brierScore function. In this example, you use the credit validation data set, which includes a table, ScorecardValidationData, that contains PD values and their corresponding default status.

Load and display the credit validation data.

load CreditValidationData.mat
head(ScorecardValidationData)

    CreditScore      PD       Default
    ___________    _______    _______

      579.86       0.14182       0   
      563.65       0.17143       0   
      549.52       0.20106       0   
      546.25       0.20845       0   
      485.34       0.37991       0   
      482.07       0.39065       0   
      579.86       0.14182       1   
      451.73         0.494       0   

Extract the variables PD and Default from the table ScorecardValidationData. Use Default as the BinaryResponse input argument.

Probability = ScorecardValidationData.PD;
BinaryResponse = ScorecardValidationData.Default;

Compute the Brier score by using the brierScore function with the fully qualified namespace risk.validation. Return a structure that contains a summary of the output metrics BrierScore, MeanResponse, and ReferenceScore.

[brierValue,Output] = risk.validation.brierScore(Probability,BinaryResponse)

brierValue = 
0.2201

Output = struct with fields:
        BrierScore: 0.2201
      MeanResponse: 0.3500
    ReferenceScore: 0.2275

Input Arguments

Probability — Probability values

numeric vector with values in the range [0,1]

Probability values, specified as a numeric vector with values in the range [0,1]. These values represent quantities such as PD estimates.

BinaryResponse — Binary response

numeric or logical vector, containing values of 1 (true) or 0 (false)

Binary response, specified as a numeric or logical vector, that contains values of 1 (true) or 0 (false). The binary response represents the target state for each value in Probability.

Output Arguments

brierValue — Brier score

numeric scalar

Brier score corresponding to the values in Probability, returned as a numeric scalar.

Output — Output metrics

structure

Output metrics, returned as a structure with the following fields:

• brierScore — Brier score

• MeanResponse — Mean value of the BinaryResponse input argument

• ReferenceScore — Brier score obtained by using the MeanResponse field

More About

Brier Score

The Brier score [1], [2] is defined as:

$$BS=\frac{1}{N}{\displaystyle \sum _{i=1}^N{(P_i-Y_i)}^2}$$

where N is the number of observations in the sample, P_i is the predicted probability for observation i, and Y_i is the observed binary outcome for observation i which has a value of 1 if the event occurred, or 0 otherwise.

The reference score is given by:

$$B{S}_{ref}=\frac{1}{N}{\displaystyle \sum _{i=1}^N{(\overline{Y}-Y_i)}^2}=\overline{Y}(1-\overline{Y})$$

where the following equation is the mean response value:

$$\overline{Y}=\frac{1}{N}{\displaystyle \sum _{i=1}^N{Y}_i}$$

P_i is typically a predicted probability of default for borrower i, and Y_i indicates whether the borrower defaulted or not.

References

[1] Brier, G. W., “Verification of Forecasts Expressed in Terms of Probability”, Monthly Weather Review, Vol. 78, Number 1, January, 1950.

[2] Basel Committee on Banking Supervision, “Studies on the Validation of Internal Rating Systems”, Working Paper 14, May, 2005. https://www.bis.org/publ/bcbs_wp14.htm.

Version History

Introduced in R2025a