trust-region reflective algorithm in lsqnonlin and fmincon: same behavior for "unconstrained" optimization"?

Say I provide constraints to lsqnonlin and fmincon. For both, I will choose trust-region-reflective algorithm

Do both fmincon and lsqnonlin behave identical in case the constraints are not active (unconstrained optimization)?

By identical, I mean is the same variant of trust-region-reflective algorithm implemented in fmincon and lsqnonlin, or are they different?

Answer 1

[Matt J]

No, they are not the same. They both take steps in a 2D subspace, but for fmincon, the basis vectors for the subspace are determined from the exact Hessian (see equations (3,4) here), whereas for lsqnonlin, they are determined from the Jacobian (see equation (6) here).

[SA-W]

How does Matlab find the exact Hessian (for fmincon) if I only supply the gradient along with the objective?

[Matt J]

How does Matlab find the exact Hessian (for fmincon) if I only supply the gradient along with the objective?

If you only supply the gradient, then fmincon's Hessian calculation will not be "exactly exact". It will compute the Hessian by finite differences. The point though, is that lsqnonlin won't even try to approximate the exact Hessian. It will always rely on the Jacobian of the residuals exclusively, which is not theoretically equivalent to a Hessian calculation except in specific cases like when your residual function is linear.

EDIT: Below is an example showing how different results can be reached by lsqnonlin and by fmincon both with an analytical and finite difference Hessian computation. Note that the final resnorm is in all cases the same.

clc,runIt(false)

 
LSQNONLIN

x = 1×3

0.4234 2.8138 2.2643

resnorm = 0.2446

 
FMINCON

x = 1×3

-5.3427 -2.9523 -3.5029

resnorm = 0.2446

 
FMINCON - ONLY GRADIENT

x = 1×3

735.0770 737.4673 736.9168

resnorm = 0.2446

function runIt(chkGrad)

 A=[0.7989
    0.9302
    0.0047
    0.6500]*[1,1,-2];
 
e=2*ones(1,4)';

x0=[0.7646    3.1550    1.5819];


disp(' ');disp('LSQNONLIN')
opts=optimoptions('lsqnonlin','Algorithm','trust-region-reflective','SpecifyObjec',true,'Display','none','CheckGrad',chkGrad);
[x,resnorm]=lsqnonlin(@resfun,x0,[],[],opts)

disp(' ');disp('FMINCON')
opts=optimoptions('fmincon','Algorithm','trust-region-reflective','SpecifyObjec',true,'Display','none',...
                           'HessianFcn','objective','CheckGrad',chkGrad);
[x,resnorm]=fmincon(@objfun,x0,[],[],[],[],[],[],[],opts)

disp(' ');disp('FMINCON - ONLY GRADIENT')
opts=optimoptions('fmincon','Algorithm','trust-region-reflective','SpecifyObjec',true,'Display','none');
[x,resnorm]=fmincon(@objfun,x0,[],[],[],[],[],[],[],opts)


    function [r,J,z]=resfun(x)
 
        x=x(:);
        z=A*x+1;
        r=z.^e./e;

        J=z.^(e-1).*A;

    end

    function [fval,grad,Hess]=objfun(x)

        [r,J,z]=resfun(x);
        fval=norm(r).^2;
        grad=2*(J.' * r);

        if nargout>2
         Hess=A.'*diag((2.*z.^(2.*e - 1))./e)*A;
        end
        
    end

end

[SA-W]

Interestingly, the three solutions differ only by a constant. Would you consider the solution "FMINCON - ONLY GRADIENT" to be the worst of the three solutions?

I obtained very good results with lsqnonlin (trust-region-reflective), but now my problem has both bounds and inequality constraints, which is why I have to switch to fmincon. If both bounds and inequality constraints are present, I can not use trust-region-reflective algorithm anymore. Which of the algorithm in fmincon resembles the trust-region-reflective algorithm most closely?

[Matt J]

Interestingly, the three solutions differ only by a constant.

Because [1;1;1] is in the null space of A.

Would you consider the solution "FMINCON - ONLY GRADIENT" to be the worst of the three solutions?

They all achieve the same resnorm so how could it be worse than any of the others?

Which of the algorithm in fmincon resembles the trust-region-reflective algorithm most closely?

I don't think a comparison can be drawn. Inequality constraints are going to change the algorithm a lot.