Question(s) regarding the output of mb.gof.test on global model.

207 views
Skip to first unread message

Christopher Dennison

unread,
Apr 9, 2024, 1:05:01 PM4/9/24
to unmarked
Good afternoon everyone,

I hope this message finds you well. I have a (few) questions regarding the output of an occupancy model I recently ran using unmarked, and the resulting mb.gof.test output using the global model.

i) The output of my Mackenzie & Bailey test (mb.gof.test) using 10000 sims yields a p-value of 0.5878 and a c-hat of 0.35. Now I understand that c-hat is a measure of over-dispersion, and that a c-hat of less than one would indicate 'under-dispersion' (variance < mean). The help page for the mb.gof.test states that c-hats of <<1 might indicate lack of fit. Wondering if it is appropriate to continue with analysis considering my c-hats value is lower than 1 (but perhaps not egregiously so). 

ii)  Following my mb.gof.test on the global model, I ran candidate models with different covariates. Two of these seemed quite 'off' to me, namely due to the fact that their AIC values were lower than all other models, but the variance and SE of my random effect in these models were MUCH higher than my other models.

iii) Lastly (sorry for the novel), I found that in running the mb.gof.test on my global model, model fitting failed for a large number of simulations.

I have a hunch that all of these issues relate to one another, but would appreciate any insight you might have. I will gladly send my code and data files for context if need be.

Thank you so, so much for your time.

- Chris

Marc J. Mazerolle

unread,
Apr 9, 2024, 8:42:27 PM4/9/24
to unma...@googlegroups.com
Hi Chris,

The short answer regarding the failure of some iterations of the
goodness of fit test is that it can happen if you have a small data set
with very high or very low beta estimates. 

Here are a few things to consider with the original data set that could
lead to problems:
-How many sites do you have in your analysis?
-What is the proportion of sites where the species was detected? 
-How many parameters do you have in your models?
-Did you get any warning messages when you ran your models with the
original data set? 
-Were any SE's in some models unusually large relative to the beta
estimates? 
-Was the variance of any random effect particularly low? 

If you can send us examples of the output of some models that could be
helpful.

Best,

Marc

--
____________________________________
Marc J. Mazerolle
Professeur agrégé et directeur du bac. en environnements naturels et
aménagés
Département des sciences du bois et de la forêt
2405 rue de la Terrasse
Université Laval
Québec, Québec G1V 0A6, Canada
Tel: (418) 656-2131 ext. 407120
Email: marc.ma...@sbf.ulaval.ca

-------- Message initial --------
De: Christopher Dennison <christophe...@gmail.com>
Répondre à: unma...@googlegroups.com
À: unmarked <unma...@googlegroups.com>
Objet: [unmarked] Question(s) regarding the output of mb.gof.test on
global model.
Date: Tue, 09 Apr 2024 10:05:01 -0700

Christopher Dennison

unread,
Apr 9, 2024, 8:59:32 PM4/9/24
to unma...@googlegroups.com
Good evening Marc,

Thank you so much for your response.

I have used a 'stacked' dataset over 3 years with 110 sites each year (2 visits each site). I use sites as a random effect in my models.
For this particular species, it was detected at only 24 visits.

My global model has two parameters for detection, and 4 for occupancy, including my random effect (site).

Here is the output for one of the two models that I deem 'suspect' due to the huge random effect variance:

Call:
occu(formula = ~1 ~ scale(build_100m) + (1 | site), data = INBU_stat.sample.unmarkedFrame_cov)

Occupancy (logit-scale):
Random effects:
 Groups        Name Variance Std.Dev.
   site (Intercept) 4231.176   65.047

Fixed effects:
                   Estimate     SE       z  P(>|z|)
(Intercept)       -12.40749 2.5128 -4.9377 7.90e-07
scale(build_100m)  -0.00561 0.0652 -0.0861 9.31e-01

Detection (logit-scale):
 Estimate     SE     z  P(>|z|)
    -1.62 0.0961 -16.8 1.49e-63

AIC: 180.7833 
Number of sites: 294
optim convergence code: 0
optim iterations: 68 
Bootstrap iterations: 0 

Compared to the output of a model which had much lower variance:

Call:
occu(formula = ~1 ~ scale(vl_100m) + scale(build_100m) + (1 | 
    site), data = INBU_stat.sample.unmarkedFrame_cov)

Occupancy (logit-scale):
Random effects:
 Groups        Name Variance Std.Dev.
   site (Intercept)    0.048    0.218

Fixed effects:
                  Estimate    SE      z  P(>|z|)
(Intercept)        -1.9245 0.645 -2.983 0.002852
scale(vl_100m)      1.1628 0.342  3.403 0.000667
scale(build_100m)   0.0869 0.317  0.274 0.784328

Detection (logit-scale):
 Estimate    SE     z P(>|z|)
    -1.21 0.545 -2.23  0.0258

AIC: 190.5375 
Number of sites: 294
optim convergence code: 0
optim iterations: 35 
Bootstrap iterations: 0 

In my original run, I simply excluded the two models with exceptionally high variance in the random effect. The results of the mb.gof.test on the global model turned out that c-hat of 0.35, which I assume I can correct to c-hat = 1.

Thank you again for your time and support. It is much appreciated.

- Chris 🐦🌱

--
*** Three hierarchical modeling email lists ***
(1) unmarked (this list): for questions specific to the R package unmarked
(2) SCR: for design and Bayesian or non-bayesian analysis of spatial capture-recapture
(3) HMecology: for everything else, especially material covered in the books by Royle & Dorazio (2008), Kéry & Schaub (2012), Kéry & Royle (2016, 2021) and Schaub & Kéry (2022)
---
You received this message because you are subscribed to a topic in the Google Groups "unmarked" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/unmarked/-EiXOCJYcVk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to unmarked+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/28d781a220b2abc55177911e0a5fb385c7047396.camel%40sbf.ulaval.ca.

Marc J. Mazerolle

unread,
Apr 9, 2024, 9:26:16 PM4/9/24
to unma...@googlegroups.com
Hi Chris,

Thanks for the additional information. Twenty-four detections across a
matrix of 294 rows and 2 columns is not a lot. An average site has a
very low occupancy: inverse logit of -12.41 = 4e-06. I suspect this is
part of the issue with the failure of iterations for some simulated
data sets in mb.gof.test( ). I notice that you did not include any
covariate on detection probability. If you have some potential
variables relevant for your species, I would add them in your model.
Adding good detection covariates generally improves the model. This
being said, I would not try to estimate many parameters in your model
with the sparse data at hand. The estimation of the random effect
potentially exacerbates the problem.

The large difference in variance between the two models is a bit
troubling. This could be due to your sites differing strongly in terms
of the vl_100m variable. Once included as an explanatory variable on
occupancy, there is not much remaining to be explained by the site
random effect.

Overall, my suggestion would be to maybe combine the data with 1 or
more similar species so that you can share some parameters between
species. Otherwise, I think the data set is too sparse to get good
estimates. Simulation would tell you how well the model can estimate
the parameters for data sets with similar characteristics as your own.
That would provide the best answer to your question. You could check
Marc Kéry and Andy Royle's Applied Hierarchical Modeling in ecology
(volume 1 and 2) for helpful advice on simulating data for different
models in unmarked.

Good luck,

Marc
--
____________________________________
Marc J. Mazerolle
Professeur agrégé et directeur du bac. en environnements naturels et
aménagés
Département des sciences du bois et de la forêt
2405 rue de la Terrasse
Université Laval
Québec, Québec G1V 0A6, Canada
Tel: (418) 656-2131 ext. 407120
Email: marc.ma...@sbf.ulaval.ca

-------- Message initial --------
De: Christopher Dennison <christophe...@gmail.com>
Répondre à: unma...@googlegroups.com
À: unma...@googlegroups.com
Objet: Re: [unmarked] Question(s) regarding the output of mb.gof.test
on global model.
Date: Tue, 09 Apr 2024 20:59:14 -0400

Christopher Dennison

unread,
Apr 10, 2024, 11:49:05 AM4/10/24
to unma...@googlegroups.com
I really appreciate your insight, Marc. Thank you! 🐦🌱

Reply all
Reply to author
Forward
0 new messages