optimization using both fminsearch and fminunc
Ka Lee
I am trying to perform an optimization problem. I believe that while fminsearch and fminunc use different algorithms, they should arrive at the same (or very similar) answer. Furthermore, one benefit of fminunc is that it gives you the estimated Hessian.
So, I first use fminsearch to obtain a solution. Then I plug in this solution as the initial value for fminunc.
However, fminunc produces an exitflag = -2, saying "Line search cannot find an acceptable point along the current search direction."
What does that mean? Should I be worried since my objective of using fminunc is only to get the estimated Hessian?
Thanks.
Ka Lee
John D'Errico
You are using the wrong tool here. Fminunc will
only compute an ESTIMATE of the hessian that
is generated by updates from the computations.
It does not explicitly compute the hessian at any
point. So use of fminunc to compute the hessian
at this point is fatuous. If you start it out at the
optimum, fminunc will never iterate from the start
point, so it will compute no estimate of any value
at all.
If your goal is to compute a hessian matrix, then
use a tool designed to estimate that matrix
accurately. My derivest tools have a Hessian matrix
estimator for a general function.
http://www.mathworks.com/matlabcentral/fileexchange/13490
HTH,
John
Ka Lee
Dear John,
Thank you for your wonderful set of programs. Actually, my goal is to be able to obtain the standard errors of the point estimates. For a function whose analytical Hessian expression is too complicated, the only way I know how to obtain the standard errors is by calling fminunc and asking it to output the approximated Hessian matrix. Then, I simply compute sqrt(diag(inv(hess))).
Is it true that I can use your program to obtain the Hessian matrix and perform the same calculation to obtain the standard errors?
Thank you,
Ka Lee
John D'Errico
> Thank you for your wonderful set of programs. Actually, my goal is to be able to obtain the standard errors of the point estimates. For a function whose analytical Hessian expression is too complicated, the only way I know how to obtain the standard errors is by calling fminunc and asking it to output the approximated Hessian matrix. Then, I simply compute sqrt(diag(inv(hess))).
>
> Is it true that I can use your program to obtain the Hessian matrix and perform the same calculation to obtain the standard errors?
>
Would I kid you? 8-)
Yes. It does compute the hessian matrix.
As I said, the estimate of the hessian that fminunc
would have returned need not be terribly accurate.
It depends on the iterations and the updates
performed.
rosen = @(xy) (1-xy(1))^2 + 100*(xy(2) - xy(1)^2)^2;
Obviously, the function is minimized at [1 1]. Fminunc
agrees with this.
>> x = fminunc(rosen,[2 2])
Warning: Gradient must be provided for trust-region method;
using line-search method instead.
> In fminunc at 265
Optimization terminated: relative infinity-norm of gradient less than options.TolFun.
x =
1 1
What do the derivest tools tell us about the
Hessian matrix at that point?
[H,err] = hessian(rosen,x)
H =
802 -400
-400 200
err =
1.5078e-12 4.0555e-11
4.0555e-11 3.7695e-13
Now, can we get the same information from fminunc?
Yes, it turns out that fminunc will compute the hessian
using finite differences at the optimum, even if only one
iteration was performed.
[X,FVAL,EXITFLAG,OUTPUT,GRAD,Hf]=fminunc(rosen,x);Hf
Warning: Gradient must be provided for trust-region method;
using line-search method instead.
> In fminunc at 265
Optimization terminated: relative infinity-norm of gradient less than options.TolFun.
Computing finite-difference Hessian using user-supplied objective function.
Hf =
802.29 -400.02
-400.02 200
I will point out that the estimate was relatively poor
compared to the derivest tool.
John
Alan Weiss
the end of its run, NOT based on its previous iterates, but simply based
on the final point. Strange, isn't it? It does seem that John's tool is
more accurate, but I believe that fminunc is often a reasonable way to
obtain a Hessian estimate at the final point.
You can even control the quality of its derivative estimate to some
extent, using the 'FinDiffType', 'DiffMinChange', and 'DiffMaxChange'
options.
Alan Weiss
MATLAB mathematical toolbox documentation
Ka Lee
Since my goal is to obtain the standard errors and I cannot derive the analytical expression for the Hessian matrix for my function, how would you suggest I proceed?
Again, I can use fminsearch to obtain the point estimates. But I also want the standard errors...Now it seems like John's tool is the answer I need.
Would you say otherwise?
Thanks,
Ka Lee
Alan Weiss <awe...@mathworks.com> wrote in message <h477iq$spp$1...@fred.mathworks.com>...
John D'Errico
> the end of its run, NOT based on its previous iterates, but simply based
> on the final point. Strange, isn't it? It does seem that John's tool is
> more accurate, but I believe that fminunc is often a reasonable way to
> obtain a Hessian estimate at the final point.
>
> You can even control the quality of its derivative estimate to some
> extent, using the 'FinDiffType', 'DiffMinChange', and 'DiffMaxChange'
> options.
Alan, do you know if it was it an older release
that used the updates from quasi-Newton
iterations to return a Hessian approximant? I
thought this was true in the past. Perhaps I
was wrong even then.
But, yes, indeed fminunc does give you a FD
Hessian as it is now. For the purposes of
parameter variance estimates, this is probably
entirely acceptable.
John
Alan Weiss
(which I have not studied, but seems to be more accurate based on the
discussion so far). I believe either will give you sufficiently accurate
Hessian estimates.
Alan Weiss
MATLAB mathematical toolbox documentation
Alan Weiss
John, you may be thinking of fmincon, which does not compute a full
Hessian at any point in its iterations, including the final point.
fminunc has not changed its behavior at least since V3.0 (R14).
I should document this behavior better. Thanks for the question.