How do I find slack variables in SVM?/ Distance to the boundary?

Hi, I used "svmtrain" to train the algorithm: svmStruct=svmtrain(xdata,group); I used "svmclassify" to classify.

My data is not perfectly linearly separable but I still used a linear classifier. In theory, allowances are made for by a slack variable. (Soft margin) I refer to the toolbox help where the theory is. It mentions the slack variable and 2 ways it is computed. http://www.mathworks.com/help/bioinfo/ug/support-vector-machines-svm.html

My issue is that, svmStruct does not save the slack variable. Neither can I find it in the function to recall it and save it.

If not, how can I find the distance from each data point to the boundary?

Can anyone help me with this? Thanks

Answer 1

[the cyclist]

It is the input parameter 'boxconstraint' to the svmtrain() command. The default value is 1.

[Mech Princess]

SVMStruct = svmtrain(Training,Group,'boxconstraint',1);

Now how do I get the values? There are no additional variables in SVMStruct when I do this? Thanks

[the cyclist]

What values? 'boxconstraint' is an input parameter. Why would it need to be reproduced in the output?

[Mech Princess]

I thought you meant that if you enable 'boxconstraint', it will give me the slack variables

I need distance from data point to the boundary. I thought Slack variables give this. It tells how much of slack is there. In the link given above, it calls it

si

[the cyclist]

Sorry! I thought the slack variable was the parameter C (which is actually the "penalty parameter").

I am not 100% sure that the slack variables have to be explicitly calculated to solve for the support vectors. (It's been a long time since I have used these techniques, and I actually don't have the toolbox at this time, to check.)

Sorry to have been more a distraction than a solution!

[Mech Princess]

Thats fine. Hope someone will answer. Thanks

Answer 2

[Ilya]

By definition, a slack variable for observation x with label y (-1 or +1) is max(0,1-y*f), where f is the SVM prediction (soft score ranging from -inf to +inf). svmclassify does not return the scores, so you need to compute the SVM scores yourself. Start with the definition of the SVM model, compute kernel products, multiply by the alpha coefficients and add the bias term. It is easier than it sounds.

[Mech Princess]

Thank you. I did it with your suggestion. But now have another issue. See what I implemented in MATLAB below;

My data is positive integers<=6. sparse-mostly zeros.

n = no of support vectors for training data
p = no of features in the input data
y = label (+1/-1) (nx1 matrix)
x = data points (nxp matrix)
w = alpha*y*x; (px1)

N = no of testing data
X = testing data (Nxp matrix)
Y = label given by SVM (Nx1 vector)
b = bias (single value but made to a Nx1 vector)
s = slack (Nx1 vector)
Y(w*X-b) >= 1- s

s >= 1-Y(w*X-b); s<0 means no slack-no alpha

My w*X is very small (10^-14) So essentially (wX-b) = -b; s = 1+Yb

Now my slack variables give me only the label. Essentially, I have found another way to interpret the label using the already known results :(

The distance to the boundary is what I want. Do you have any suggestions what I am doing wrong? (I know this might not be a MATLAB problem anymore) Thanks again

[Ilya]

What you posted does not make much sense.

alpha is undefined.

If alpha is n-by-1, y is n-by-1 and x is n-by-p, alpha*y*x gives an error.

There is no such thing as "slack variable" for test data. Slack is used for solving the SVM problem and makes sense for training data only.

Y(w*X-b) is not a valid MATLAB expression.

[Mech Princess]

Thanks. Sorry. alpha is whats given from the SVMStruct and I used dot product to get w (p-by-1). Should w be (p-by-1)?

Anyway, I realized I cant find what I want with slack variables. Thank you. Do you know how to find the distance from the boundary for testing data?

Thanks

[Ilya]

Here is how you compute SVM scores for the new data in Xnew and SVM model saved in svm_struct:

sv = svm_struct.SupportVectors;
alphaHat = svm_struct.Alpha;
bias = svm_struct.Bias;
kfun = svm_struct.KernelFunction;
kfunargs = svm_struct.KernelFunctionArgs;
f = kfun(sv,Xnew,kfunargs{:})'*alphaHat(:) + bias;

The distance from the boundary depends on what you mean by "boundary". The decision boundary is defined by f=0, and the signed distance is then f. The support hyperplanes are defined by y*f=1, and the signed distance is y*f-1.

[Mark]

Thanks!

Only one remark: I think that this works fine if 'autoscale' is set to false (in the svmtrain function). If the data is scaled, you should also scale Xnew before you feed it to the kernel function:

shift = svm_struct.ScaleData.shift;
scale = svm_struct.ScaleData.scaleFactor;

Now you can scale Xnew:

XnewScaled = ( Xnew - shift ) .* scale;

and then use XnewScale in the kernel function as above:

f = kfun( sv, XnewScaled, kfunargs{:} )' * alphaHat + bias

[Aliza Rubenstein]

Thanks for this information. I think it should be Xnew + shift, not Xnew - shift. shift is the negative of the mean.