Finding best parameters of SVM

Hi.

I’m designing a system that I can optimize parameters of a support vector machine (SVM) with genetic algorithm, harmony search and another optimization algorithms to find the best structure of SVM for a specific data. My problem is binary classification with 0 and 1 output and I normalize data (mapmaxmin o mapstd) before insert it to system. Besides it in some cases I use dimension reduction (for example FDA) to reduce my features. For this normalized data I must set the boundary of searching space in optimization algorithm. This is my SVM function:

svmstruct=svmtrain(TrainInputs,TrainTargets,...
'boxconstraint',Penalty,...
'kernel_function','rbf','method','QP',...
'rbf_sigma',Sigma,...
'autoscale','false');

I optimize only 'boxconstraint' and ‘rbf sigma’. For boxconstraint, my algorithm is searching in [0.001 400] and for sigma the searching space is same [0.001 400]. IS this searching boundaries is suitable for my problem or I must change these boundaries? Otherwise, I set ‘autoscale’ to ‘false’. Which one is better in my problem? ‘false’ or ‘true’ ?

I set kernel function to rbf. is this a good approach for this problem?

Thanks.

Answer 1

[Ilya]

Plot the classification error (or whatever it is you are optimizing) against the two optimized parameters. Is the minimum well inside the box defined by your bounds? If yes, you likely found the global minimum.

If you set 'autoscale' to false, the SVM model is more sensitive to variables with large variance and less sensitive to variables with small variance. Is this what you want? Or would you rather treat your variables on equal footing, no matter how their variance values compare? If the latter, set 'autoscale' to true.

In my experience, when variables are standardized, the optimal kernel width is often close to 1.

To optimize the box constraint, I would normally use a uniformly spaced logarithmic scale, say from 1e-6 to 1e+6.

[Jack]

Thank you for your answer Ilya.

You said your suggestion range for box constraint is [1r-6 1e+6]. What is your suggestion range for sigma of rbf?

That was a good approach to plot classification error against the two optimized parameters but I must set a boundary in general because I’m designing a gui so I must set these boundaries manually in codes.

Another question : Do you think RBF is good in my binary classification problem or I must check other kernel functions too?

Thanks.

[Ilya]

What is your suggestion range for sigma of rbf? That would depend on how many variables you have. Take a sufficiently broad interval around 1 on the logarithmic scale.

but I must set a boundary in general There is no such thing as boundaries that would work for any dataset.

RBF is the one most commonly used for high accuracy. There is no guarantee some other kernel would not outperform RBF on a specific dataset. The answer depends on what kind of accuracy improvement you would care about and how much time/effort you want to invest.

[Jack]

Thank you again.

My system select best inputs (besides parameters of SVM) so I don’t have specific number of variables and it is changing in every iteration. But my initial inputs variables are 21. (So my input variables are between 1 and 21).

So I have to set this boundary in my system. According to what was explained what is your opinion in this case?

Thanks.

[Ilya]

OK, let me be more specific. The optimal value would also depend on the kind of predictors you have (categorical, discrete, continuous), their correlations with the class label, and their mutual correlations.

If I were in your shoes, I would just play with as much data pertinent to my specific analysis as I can to see what works best for me. Asking such a specific question to people who never looked at your data won't take you far.

Answer 2

[chit paing]

rather good solution