Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
fmincon -> Hessian -> Covariance matrix
163 views
Skip to first unread message
Céldor
Feb 10, 2012, 6:05:27 AM2/10/12
to
Dear All,
I've got to calculate a covariance matrix from my data set. My objective function is based on log(mormcdf()) - log of standatd normal cumulative probability. I need to find Hessian and then information matrix which AFAIK is the covariance matrix.
If I am correct, I would need to obtain Hessian, then inverse its an, and multiply by -1 which is information matrix and then the inf. matrix is simply the covariance matrix if I calculate it for the estimated parameters.
Can someone please help me which steps I need to take? I though fmincon which I use for finding estimates can return a Hessian but I've read this Hessian may not be accurate.
Thank you for any help.
I've got to calculate a covariance matrix from my data set. My objective function is based on log(mormcdf()) - log of standatd normal cumulative probability. I need to find Hessian and then information matrix which AFAIK is the covariance matrix.
If I am correct, I would need to obtain Hessian, then inverse its an, and multiply by -1 which is information matrix and then the inf. matrix is simply the covariance matrix if I calculate it for the estimated parameters.
Can someone please help me which steps I need to take? I though fmincon which I use for finding estimates can return a Hessian but I've read this Hessian may not be accurate.
Thank you for any help.
Alan Weiss
Feb 10, 2012, 9:53:55 AM2/10/12
to
can get a reliable Hessian estimate:
http://www.mathworks.com/help/toolbox/optim/ug/bsapd3i.html
If you use fmincon because you have constraints, then I am not sure that
the Hessian relates well to the covariance matrix, because of what
happens at the active constraints. But I am not an expert in this area,
so I could be wrong.
Alan Weiss
MATLAB mathematical toolbox documentation
Céldor
Feb 10, 2012, 11:54:11 AM2/10/12
to
"Céldor " <ze...@op.pl> wrote in message <jh2tln$6c1$1...@newscl01ah.mathworks.com>...
Hello Alan.
Thanks for the answer. When I read hessian from fminunc--this hessian is similar to the one returned from fmincon, I get negative numbers. I think it is not right. When the Hessian is all positive, sometimes when I calculate its inverse it gets also negative numbers. It is like Covariance contains negative numbers. What am I doing wrong and how to handle this problem?
Thanks for the answer. When I read hessian from fminunc--this hessian is similar to the one returned from fmincon, I get negative numbers. I think it is not right. When the Hessian is all positive, sometimes when I calculate its inverse it gets also negative numbers. It is like Covariance contains negative numbers. What am I doing wrong and how to handle this problem?
Matt J
Feb 10, 2012, 12:06:11 PM2/10/12
to
"Céldor " <ze...@op.pl> wrote in message <jh3i3j$fdf$1...@newscl01ah.mathworks.com>...
You probably need to compute the Hessian analytically in order for this application to work well. The numerical approximations used by FMINUNC will be subject to various kinds of inaccuracies.
Incidentally, though, there's no reason the Hessian (or it's inverse) shouldn't contain negative numbers. If your minimum has a strictly positive definite Hessian, then the diagonals should be >0, but nowhere else in the matrix is this guaranteed, and moreover that's only if you've converged sufficiently closely to the minimum.
>
> Thanks for the answer. When I read hessian from fminunc--this hessian is similar to the one returned from fmincon, I get negative numbers. I think it is not right. When the Hessian is all positive, sometimes when I calculate its inverse it gets also negative numbers. It is like Covariance contains negative numbers. What am I doing wrong and how to handle this problem?
=================
> Thanks for the answer. When I read hessian from fminunc--this hessian is similar to the one returned from fmincon, I get negative numbers. I think it is not right. When the Hessian is all positive, sometimes when I calculate its inverse it gets also negative numbers. It is like Covariance contains negative numbers. What am I doing wrong and how to handle this problem?
You probably need to compute the Hessian analytically in order for this application to work well. The numerical approximations used by FMINUNC will be subject to various kinds of inaccuracies.
Incidentally, though, there's no reason the Hessian (or it's inverse) shouldn't contain negative numbers. If your minimum has a strictly positive definite Hessian, then the diagonals should be >0, but nowhere else in the matrix is this guaranteed, and moreover that's only if you've converged sufficiently closely to the minimum.
Céldor
Feb 12, 2012, 11:01:10 AM2/12/12
to
"Céldor " <ze...@op.pl> wrote in message <jh2tln$6c1$1...@newscl01ah.mathworks.com>...
Matt,
Thank you for your answer. I tried to calculate hessian but I am afraid I have stuck. I know the idea behind theory. I wouldn't have problem to analytically calculate hessian of any other function but my log-likelihood is based on normal cdf. I found that this function can be calculated from the formula below (I use log of this function):
log(f(y;mu,sd)) = log[ 1/2*[1 + erf((y - mu)/sqrt(2*sd^2))] ]
Now, although I am trying to find out what the gradient and hessian over b0 b and sd variables is, I still feel like I have stuck. If you have any idea I would be very grateful for any help on that :)
Regards.
Thank you for your answer. I tried to calculate hessian but I am afraid I have stuck. I know the idea behind theory. I wouldn't have problem to analytically calculate hessian of any other function but my log-likelihood is based on normal cdf. I found that this function can be calculated from the formula below (I use log of this function):
log(f(y;mu,sd)) = log[ 1/2*[1 + erf((y - mu)/sqrt(2*sd^2))] ]
Now, although I am trying to find out what the gradient and hessian over b0 b and sd variables is, I still feel like I have stuck. If you have any idea I would be very grateful for any help on that :)
Regards.
Matt J
Feb 12, 2012, 11:28:10 AM2/12/12
to
"Céldor " <ze...@op.pl> wrote in message <jh8no6$i4l$1...@newscl01ah.mathworks.com>...
There are no bo,b variables in the expression you've shown. But aside from that, what's the problem? You just need to take derivatives and use the chain rule a few times. The derivative of erf is shown here
http://en.wikipedia.org/wiki/Error_function#Taylor_series
>
> log(f(y;mu,sd)) = log[ 1/2*[1 + erf((y - mu)/sqrt(2*sd^2))] ]
>
> Now, although I am trying to find out what the gradient and hessian over b0 b and sd variables is, I still feel like I have stuck. If you have any idea I would be very grateful for any help on that :)
================
> log(f(y;mu,sd)) = log[ 1/2*[1 + erf((y - mu)/sqrt(2*sd^2))] ]
>
> Now, although I am trying to find out what the gradient and hessian over b0 b and sd variables is, I still feel like I have stuck. If you have any idea I would be very grateful for any help on that :)
There are no bo,b variables in the expression you've shown. But aside from that, what's the problem? You just need to take derivatives and use the chain rule a few times. The derivative of erf is shown here
http://en.wikipedia.org/wiki/Error_function#Taylor_series
Céldor
Feb 12, 2012, 1:14:09 PM2/12/12
to
"Matt J" wrote in message <jh8paq$mf9$1...@newscl01ah.mathworks.com>...
> But aside from that, what's the problem? You just need to take derivatives and use the chain rule a few times. The derivative of erf is shown here
>
> http://en.wikipedia.org/wiki/Error_function#Taylor_series
There is no problem. I was trying to make sure I am taking the proper derivative of ln(erf(.)). Thanks for the link. I have found it already and I started to work with it.
Regards
> "Céldor " <ze...@op.pl> wrote in message <jh8no6$i4l$1...@newscl01ah.mathworks.com>...
> >
> > log(f(y;mu,sd)) = log[ 1/2*[1 + erf((y - mu)/sqrt(2*sd^2))] ]
> >
> > Now, although I am trying to find out what the gradient and hessian over b0 b and sd variables is, I still feel like I have stuck. If you have any idea I would be very grateful for any help on that :)
> ================
>
> There are no bo,b variables in the expression you've shown.
mu = b0 + x*b.
> >
> > log(f(y;mu,sd)) = log[ 1/2*[1 + erf((y - mu)/sqrt(2*sd^2))] ]
> >
> > Now, although I am trying to find out what the gradient and hessian over b0 b and sd variables is, I still feel like I have stuck. If you have any idea I would be very grateful for any help on that :)
> ================
>
> There are no bo,b variables in the expression you've shown.
> But aside from that, what's the problem? You just need to take derivatives and use the chain rule a few times. The derivative of erf is shown here
>
> http://en.wikipedia.org/wiki/Error_function#Taylor_series
Regards
Alan Weiss
Feb 13, 2012, 8:55:27 AM2/13/12
to
http://www.mathworks.com/help/toolbox/optim/ug/brn4nh7.html#brv_i_1
Céldor
Feb 14, 2012, 10:04:49 AM2/14/12
to
> If you have Symbolic Math Toolbox, let it do the work for you:
> http://www.mathworks.com/help/toolbox/optim/ug/brn4nh7.html#brv_i_1
>
> Alan Weiss
> MATLAB mathematical toolbox documentation
Hi Alan,
> http://www.mathworks.com/help/toolbox/optim/ug/brn4nh7.html#brv_i_1
>
> Alan Weiss
> MATLAB mathematical toolbox documentation
This is most useful. Thank you for that :)
0 new messages