SVMTRAIN - LEAST SQUARE METHOD OR QUADRATIC PROGRAMMING METHOD

Hi,

I am using svmtrain function. QP method is really slow (too much) and sometimes it crashes. When I use LS (least square) method it is SUPER fast.

Could you explain me what is the difference btw these two methods. In theory do they have to give the same result ? why one is much faster than the other method ?

Thank you for your help.

Answer 1

[Ilya]

LS minimizes mean squared error to solve the SVM regression problem. In SVM classification, coefficients alpha are bounded between 0 and C (the box constraint). Support vectors (observations with positive alphas) obtained by solving the classification SVM problem are a fraction of the training set. For minimization of MSE, coefficients alpha are not bounded, and most usually every observation in the training set gets a non-zero alpha coefficient.

QP solves the classification problem but is slow on large datasets. For large-scale classification problems, your best bet is SMO.

Answer 2

[Shashank Prasanna]

QP solves the quadratic optimization using the Hessian. In a quick sentence, the advantage is that it is higher in precision and the disadvantage is that it doesn't scale for large problems (large training data).

LS just solves a linear system using the \ operator for a least square fit. There may be precision difference between the two but both are good and depends on the nature of the problem.

Take a look at the following page in the documentation:

http://www.mathworks.com/help/bioinfo/ug/support-vector-machines-svm.html#bs3tbev-7

Also feel free to take a look at the code by:

>> edit svmtrain