risk.validation.binomialTest

Return binomial test result

Since R2025a

Syntax

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials)

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,Name=Value)

[hBinTest,binTestOutput] = risk.validation.binomialTest(___)

Description

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials) returns the binomial test result, hBinTest, for a given set of probabilities, events, and trials. The output is 1 if the test rejects the null hypothesis at the 95% confidence level, or 0 otherwise. Probability contains numeric values that represent quantities such as probability of default (PD) estimates.

hBinTest = risk.validation.binomialTest(Probability,NumEvents,NumTrials,Name=Value) specifies optional name-value arguments. For example, you can set a specific confidence level for the binomial test by using the ConfidenceLevel name-value argument.

[hBinTest,binTestOutput] = risk.validation.binomialTest(___) also returns a structure binTestOutput, that contains the method used in the binomial test and a table of summary metrics.

Examples

Apply Binomial Test on Probability of Default Data

Use the binomialTest function to determine whether the binomial test rejects the null hypothesis for a 0.95 confidence level in a probability of default (PD) data set. In this example, you use the credit validation data set, which includes a table, ScorecardValidationData, that contains PD values and their corresponding default status. Before you apply the test, it is common practice to:

• Group the probabilities by deciles.

• Compute the average probability of each group.

• Compute the total number of defaults and loans of each group.

After performing these steps, you then apply the test on each of the groups.

Load and display the credit validation data.

load CreditValidationData.mat
head(ScorecardValidationData)

    CreditScore      PD       Default
    ___________    _______    _______

      579.86       0.14182       0   
      563.65       0.17143       0   
      549.52       0.20106       0   
      546.25       0.20845       0   
      485.34       0.37991       0   
      482.07       0.39065       0   
      579.86       0.14182       1   
      451.73         0.494       0   

Group Probabilities by Deciles

Extract the variable, PD, from the table ScorecardValidationData and group the probabilities by using the risk.validation.groupNumberByQuantile function with the fully qualified namespace risk.validation. Specify "deciles" as the quantile type.

Probability = ScorecardValidationData.PD;
QuantileType = "deciles";
PDDecileNumber = risk.validation.groupNumberByQuantile(Probability,QuantileType);

Determine Average Probabilities by Group

Next, calculate the average probability for each group by using the groupsummary function.

PDAvgByDecile = groupsummary(Probability,PDDecileNumber,"mean");

Compute Total Defaults and Loans by Group

Extract the variable Default from the table ScorecardValidationData and use this variable as the default indicator. Then use the groupsummary function to compute the total number of defaults and loans for each group.

DefaultIndicator = ScorecardValidationData.Default;
[NumDefaultsByDecile,~,NumLoansByDecile] = groupsummary(DefaultIndicator,PDDecileNumber,"sum");

Apply Binomial Test

You can then apply the binomialTest function with the fully qualified namespace risk.validation for each group to see if the test rejects the null hypothesis. Use the average probabilities and the total number of defaults and loans as input arguments. Then, display a table with the results.

[hBinTest,binTestOutput] = risk.validation.binomialTest(PDAvgByDecile,NumDefaultsByDecile,NumLoansByDecile)

hBinTest = 10×1 logical array

   0
   1
   0
   0
   0
   0
   0
   0
   0
   0

binTestOutput = struct with fields:
     Method: "exact"
    Results: [10×8 table]

disp(binTestOutput.Results)

    RejectBinTest    NumEvents    CriticalValue     PValue     ConfidenceLevel    NumTrials    Probability    ObservedProbability
    _____________    _________    _____________    ________    _______________    _________    ___________    ___________________

        false            7              8            0.1099         0.95             36          0.13683            0.19444      
        true            11             11          0.039577         0.95             36          0.19799            0.30556      
        false            8             12           0.35964         0.95             34          0.22656            0.23529      
        false           10             11          0.060747         0.95             27          0.25552            0.37037      
        false           16             17          0.067459         0.95             42           0.2868            0.38095      
        false           13             18           0.47796         0.95             41          0.32664            0.31707      
        false           10             18           0.85307         0.95             36          0.37627            0.27778      
        false           11             16           0.52405         0.95             29          0.40319            0.37931      
        false           18             23           0.46153         0.95             40          0.45525               0.45      
        false           22             27           0.42988         0.95             39          0.56233             0.5641      

Input Arguments

Probability — Probability values

numeric vector with values in the range (0,1)

Probability values, specified as a numeric vector with values in the range (0,1). Probability contains values that indicate quantities such as PD estimates.

NumEvents — Number of events observed

numeric vector of nonnegative integers

Number of events observed, specified as a numeric vector of nonnegative integers. For PD models, NumEvents contains the number of defaults observed.

NumTrials — Number of trials

numeric vector of positive integers

Number of trials, specified as a numeric vector of positive integers. Each element in NumTrials must be greater than or equal to the corresponding elements of NumEvents. For PD models, NumTrials contains the number of loans.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: hBinTest = risk.validation.binomialTest(PDAvgByDecile,NumDefaultsByDecile,NumLoansByDecile,ConfidenceLevel=0.96) specifies a confidence level of 0.96.

ConfidenceLevel — Confidence level

0.95 (default) | numeric scalar in the range (0,1)

Confidence level of the hypothesis test, specified as a numeric scalar in the range (0,1).

Method — Hypothesis test method

"exact" (default) | "approximate"

Hypothesis test method, specified as either "approximate" or "exact".

• "exact" — binomialTest uses an exact probability based on a binomial distribution.

• "approximate" — binomialTest uses an approximate test based on a normal distribution.

Output Arguments

hBinTest — Hypothesis test result

logical vector containing values of 1 or 0

Hypothesis test result, returned as a logical vector.

• A value of 1 rejects the null hypothesis at the specified confidence level.

• A value of 0 fails to reject the null hypothesis at the specified confidence. level.

binTestOutput — Output metrics

structure

Output metrics, returned as a structure with the following fields:

• Method — Method used for the hypothesis test.

• Results — Table with columns:

  • RejectBinTest — Logical vector that indicates whether the null hypothesis was rejected. This column represents the same values as hBinTest.

  • NumEvents — Number of events observed. This column is the test statistic for the hypothesis test.

  • CriticalValue — Minimum number of events at which the test rejects the null hypothesis for the given probability and confidence level.

  • PValue — p-value for the hypothesis test returned as a scalar in the range [0,1]. A small value indicates that the null hypothesis might not be valid.

  • ConfidenceLevel — Confidence level for the hypothesis test.

  • NumTrials — Number of trials.

  • Probability — Probability values.

  • ObservedProbability — Ratio of NumEvents to NumTrials.

More About

Binomial Test

The binomial test [1] requires you to input the predicted probability P, the number of trials N, and the number of events observed, E. The null hypothesis is that the probability P is a correct estimate of the event frequency. The risk.validation.binomialTest function implements a one-sided version of the test, where the alternative hypothesis is that P underestimates the event frequency. Rejection of the binomial test means that P is statistically small compared to the observed event frequency E/N. The test statistic for the binomial test is the number of events E.

There are two versions of the test, the “exact” method, and the “approximate” method. In the “exact” method, the critical value is given by:

$$E^{*}=min\left\{k|{\displaystyle \sum _{i=k}^N\left(\begin{array}{c}N\\ i\end{array}\right)P^i{(1-P)}^{(N-i)}\le 1-\alpha }\right\}$$

where α is the test confidence level that has a default value of 95%. The critical value E* is the minimum integer value k such that the tail probability of the binomial distribution for values k and above, is less than or equal to the complement of the confidence level, 1-α. Since this is a discrete distribution, there is a positive probability that the test statistic is exactly equal to the critical value. The null hypothesis is rejected whenever the test statistic is greater than or equal to the critical value. In the “approximate” method, using the central limit theorem, the critical value is given by:

$$E_{approx}^{{}^{*}}=NP+{\Phi }^{-1}(\alpha )\sqrt{NP(1-P)}$$

where Φ is the standard normal cumulative distribution function. If the inputs to risk.validation.binomialTest are vectors of size G, then G statistical tests are run and the function returns the results for each of these tests.

For credit risk applications, P is typically a PD, N is the number of loans, and E the number of defaults. The G groups could correspond to different credit ratings or deciles based on the PD values of a continuous PD model. For market risk applications, P could represent the complement of the VaR level, N could represent the number of days in a test window such as 250 trading days, and E could represent the number of VaR exceptions.

References

[1] Basel Committee on Banking Supervision, “Studies on the Validation of Internal Rating Systems”, Working Paper 14, May, 2005. https://www.bis.org/publ/bcbs_wp14.htm.

Version History

Introduced in R2025a