validating a mixed model with random effects

[Kevin Welch]

Hi All,

Does anyone have any experience with validating a mixed model in glmmadmb?I want to check the accuracy of predictions against a new data set.  I understand using the predict function but because my model has random effects, accounting for the uncertainty in the random effects to get accurate confidence intervals is tricky.

Currently the glmmadmb package does not have the capability to predict new cases with accurate variance estimates.

Any suggestions or leads on code?

thanks!

Kevin

[Guillaume Théroux Rancourt]

Hi Kevin,

I don't have experience with the glmmadmb package, but you should check out the AICcmodavg package. It has functions for several packages to test againt a new dataset, get predicted values and SE, compare models, etc. It seems to work with glmmadmb (see here).

HTH,

Guillaume

--
Check out our R resources at http://www.noamross.net/davis-r-users-group.html
---
You received this message because you are subscribed to the Google Groups "Davis R Users' Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to davis-rug+...@googlegroups.com.
Visit this group at http://groups.google.com/group/davis-rug.
For more options, visit https://groups.google.com/d/optout.

[Jaime Ashander]

Hi Kevin,

No experience doing this with glmmADMB, but some general comments and a possible solution below.

As you pointed out, the problem of how to deal with prediction and random effects is tricky -- you must choose which RE to include in your prediction, and which are appropriate depends on the nature of the new data. An example of the effects for lme4 predict, using re.form argument, is shown in the 'customized prediction' section of my post here; http://www.ashander.info/posts/2015/04/D-RUG-mixed-effects-viz/

But, as you pointed out, the predict method in glmmADMB can't accommodate random effects. 

I think adding draws from your RE distribution to the appropriate rows of your predicted data (on the link scale), given the RE structure, would produce something similar to lme4s predict, but this would take some time to implement!

Perhaps a better, but maybe unsatisfying, alternative is to refit the model using MCMCglmm and use it to predict. I would feel ok about doing so if parameter estimates are similar, possibly computationally painful to fit with MCMC. Some worked examples of models using both packages are here: https://rpubs.com/bbolker/glmmchapter

Good luck,
Jaime

PS A further resource, if you haven't already seen, is the GLMM faq http://glmm.wikidot.com/faq but it doesn't seem to have any examples of prediction accounting for RE with glmmadmb. 

[Kevin Welch]

Thanks for the responses! Both are very helpful leads.

On Tue, Sep 15, 2015 at 11:09 AM, Guillaume Théroux Rancourt <gtran...@gmail.com> wrote:

[Myfanwy Johnston]

Hi Kevin,

For a Bayesian approach, Richard McElreath's rethinking package (which works with rstan) is another option - he has a few convenience functions to generate predictions from any models that you've fit (I've fit multilevel mixed models with the package and generated predictions to test against the original dataset, so I imagine what you're trying to do is quite possible).  Package user manual here, forthcoming book here.

Best,

Myfanwy

Myfanwy Johnston

Ph.D Candidate, Animal Behavior
University of California at Davis

Biotelemetry Lab