[ADMB Users] how reliable are ADMB estimates of variances?

[Saang-Yoon Hyun]

Hello.

A simple ADMB program of mine does an excellent job of calculating MLEs of parameters. However, its estimation of variances of parameter estimates does not look good. If someone has ever evaluated ADMB performance in terms of "variance estimation", please share your findings/experiences/opinions about the performance with me. I wonder about how reliably ADMB calculates variance estimates.

Thank you,

Saang-Yoon

_______________________________________________
Users mailing list
Us...@admb-project.org
http://lists.admb-project.org/mailman/listinfo/users

[Richard...@noaa.gov]

Generally we find ADMB's estimates of variance from the inverse
Hessian method to be sufficiently accurate. Even with large and
complex models with hundreds of estimated parameters, we find these
variance estimates to be similar to those calculated from MCMC and
from likelihood profiles.

Rick Methot

[dave fournier]

Hi,

I think the title you give to this question shows that you are
thinking about it wrong. You say "ADMB estimates of variance"
which is not what they are. What they are is the use of the Hessian of
the log-likelihood to produce variance estimates. This is not an
ADMB construct but simply a standard tool to produce such estimates.
As such the ADMB estimates produced this way are just as good as any
other scheme which produces such estimates *UNLESS* they are not
calculated correctly which would make them an ADMB bug issue.

Now it is well known that the estimates provided this way are good in
the "approximately normal" situation, but they can be very bad
in other situations. That is why there are other methods such as
profile likelihoods proveded which may be more accurate in such cases.

I believe this is all discussed in the old manual.

Dave

--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com

[Ian Taylor]

On the subject of variances (or covariances), I've just looked for the first time at the empirical covariance matrix written to the .ecm file at the end of an MCMC run by making use of the R function readBin described by Anders in his talk about MCMC available at http://admb-project.org/documentation/courses/july-2009.

As noted in the previous emails, the Hessian provides good estimates in many cases, but for those situations which seem to not be approximately normal, I'm wondering if ADMB could be extended to allow this ECM to be substituted for the Hessian as the basis for the jump function in MCMC. Obviously such an option would require repeated runs of MCMC to first estimate the ECM and then make use of it, but it seems that for some cases, the improvement in the jump function would allow faster convergence, especially for cases where the MCMC might jump between alternative states that aren't well described by the Hessian. Or perhaps I'm misinterpreting what the ECM file is or is intended for.

For anyone interested, here is the R code I used to read the ECM (where filename is a string giving the location of the .ecm file produced by ADMB at the end of an MCMC run).

# read empirical covariance file (n is upper limit on number of values)
ecmfile <- readBin(filename,what=numeric(),n=1e6)
npars <- sqrt(length(ecmfile))
ecm <- matrix(ecmfile,byrow=T,npars,npars)

-Ian