Optimizing percentage split of CNN training/test data

I am attempting to use Bayesian optimization to find the optimal percentage split between training and validation data, when training a particular CNN.

First I define my optimizable variable

percentagechange = optimizableVariable('percentage_split',[0 1],'Type','real');

Then I use the bayesopt function to optimize my objective function

ObjFcn = makeObjFcn(lgraphlayers,ds,options);

BayesObject = bayesopt(ObjFcn,percentagechange, ...
    'MaxTime',2*60*60, ...
    'IsObjectiveDeterministic',false, ...
    'UseParallel',false);

where I am calling my objective function that is defined as follows

function ObjFcn = makeObjFcn(lgraphlayers,ds,options)
ObjFcn = @valErrorFun;
    function [valError,cons,fileName] = valErrorFun(percentagesplit)
        
        [dsTrain,dsVal] = splitEachLabel(ds,percentagesplit,'randomize');
        
        [trainedNetwork, info] = trainNetwork(dsTrain,lgraphlayers,options);
              
        [predictedClass,probs] = classify(trainedNetwork,dsVal);
        valError = 1 - mean(predictedClass == dsVal.Label);
        
        fileName = num2str(valError) + ".mat";
        save(fileName,'trainedNetwork','validationError','opts')
        cons = [];
        
    end
end

The problem is that using the optimizable variable in this way causes the following error:

Error using matlab.io.datastore.ImageDatastore/splitEachLabel (line 211)

Invalid parameter type. Splitting proportions must be numeric."

Error in makeObjFcn/valErrorFun

[dsTrain,dsVal] = splitEachLabel(ds,percentagesplit,'randomize');

Error in BayesianOptimization/callObjNormally (line 2560)

[Objective, ConstraintViolations, UserData] =

this.ObjectiveFcn(conditionalizeX(this, X));

Error in BayesianOptimization/callObjFcn (line 467)

= callObjNormally(this, X);

Error in BayesianOptimization/runSerial (line 1989)

ObjectiveFcnObjectiveEvaluationTime, ObjectiveNargout] =

callObjFcn(this, this.XNext);

Error in BayesianOptimization/run (line 1941)

this = runSerial(this);

Error in BayesianOptimization (line 457)

this = run(this);

Error in bayesopt (line 323)

Results = BayesianOptimization(Options);

Error in noise_hyperparameter_optimisation (line 68)

BayesObject = bayesopt(ObjFcn,percentagechange, ...

Clearly I am not permitted to put an optimizableVariable straight in to the splitEachLabel function, but I do not know how else to approach the problem. Any help would be greatly appreciated.

Answer 1

[Raunak Gupta]

Hi Holly,

Since the percentagechange is accurately defined in the code for Bayesian optimization I don’t see any issue there. Only when looking into a similar example of Deep Learning using Bayesian Optimization and following syntax, I see that the percentagechange is optimizableVariable type and will through error in splitEachLabel while passing. From the syntax in the example I think

percentagechange.percentage_split 

is the correct argument for splitEachLabel function. Also input to valErrorFun must be percentagechange rather than percentagesplit because no such variable (percenatagesplit) exist.

You may check the same and follow the same example syntax if there is something missing in your code.

Hope this helps!