Information loss of the Mahalanobis distance in high dimensions: Matlab implementation

Information loss estimation to set a lower limit on classification rate.

The Mahalanobis distance between a pattern measurement vector of dimensionality D and the center of the class it belongs to is distributed as a chi^2 with D degrees of freedom, when an infinite training set is used. However, the distribution of Mahalanobis distance becomes either Fisher or Beta depending on whether cross-validation or re-substitution is used for parameter estimation in finite training sets. The total variation between chi^2 and Fisher as well as between chi^2 and Beta allows us to measure the information loss in high dimensions. The information loss is exploited then to set a lower limit for the correct classification rate achieved by the Bayes classifier that is used in subset feature selection.

Installation:
-------------

The 5 functions should be in the current path of Matlab.

Usage:
------

LowCCRLimit = LowCCRLimitInfLoss(D, CCR, NDc, CClasses, ErrorEstMethod)

% D: Dimensionality of the vector (2,3,4,5,...)
% CCR: The Correct Classification rate in [1/CClasses,1] (e.g. 0.8)
% NDc: The number of training samples per class (>D+1)
% CClasses: The number of classes in your problem (2,3,4,...)
% ErrorEstMethod: "Resub" for resubstitution
% "Cross" for cross-validation

Example:
--------
LowCCRLimitInfLoss(5, 0.75, 100, 5, 'Cross')

ans = 0.7288

References:
-----------

[1] Dimitrios Ververidis and Constantine Kotropoulos, "Information loss of the Mahalanobis distance in high dimensions: Application to feature selection,"

IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 12, pp. 2275-2281, 2009.

[2] Jeffrey, Knuth, "On the Lambert W Function", Advances in Computational Mathematics, volume 5, 1996, pp. 329-359.

Special thanks to Dr. Pascal Getreuer for implementing the lambertw2 function from Jeffrey Knuth publication.