understanding parboot model fit output

[Jenna McDermott]

Hello all,

I am working through some occupancy models and have been using parboot to assess model fit.

pb <- parboot(MAWA.elev, statistic=chisq, nsim=100, parallel=FALSE)

plot(pb)

From what I understand, if the model is a good fit, then the t0 under the Parametric Bootstrap Statistics should fall within the range of numbers under the t_B quantiles (which when plotted would appear as the dotted line laying within the histogram), and p-value should be >0.05 to fail to reject the null and conclude that the model fits the data well. Please correct me if I'm wrong about any of that.

I've been getting some outputs that seem conflicting, like the one I show below. The t0 does not lay within the t_B quantile range, and yet the p-value is very high, nearly 1. This indicates to me that the model does not fit well (t0 falls outside the range) and does fit well (p-value is high). I've had this occur for occupancy models of a few different bird species.

If anyone has an explanation of why this happens and what it means, I'd greatly appreciate it.

Thanks!

Jenna

Call: parboot(object = MAWA.elev, statistic = chisq, nsim = 100, parallel = FALSE)

Parametric Bootstrap Statistics:

        t0 mean(t0 - t_B) StdDev(t0 - t_B) Pr(t_B > t0)

Chisq 3876          -1572              599         0.99

t_B quantiles:

        0% 2.5%  25%  50%  75% 97.5% 100%

Chisq 4267 4715 5074 5411 5680  6647 8707

t0 = Original statistic computed from data

t_B = Vector of bootstrap samples

[John C]

Hi Jenna,

If my quick read of this is correct, the empirical statistic falls along the left tail of the bootstrap samples (i.e., t_B tends to be larger than t0). This seems to indicate underdispersion (i.e., a an estimate of c-hat < 1). 

Although I don't know that the general pattern will change, you might try AICcmodavg's 'mb.gof.test' function, particularly if you've been implementing parboot without aggregating across cells in the detection history in some way. 

More generally, I don't know that there is clear consensus regarding what to do in the case of underdispersion. Might be fine ignoring underdispersion if c-hat is close to 1, but very small values may indicate different types of problems with the model or the sampling scheme. 

John

[Jenna McDermott]

Hi John,

Yes you understood perfectly what I was trying to explain. Thanks for your response. It has given me a great starting point to do some more reading to get a better understanding of over- and under-dispersion and how to deal with them.

Would you mind explaining what you meant by "aggregating across cells in the detection history", and what the implications of doing or not doing that may be? Sorry if that is a basic question: I have only just delved into occupancy analyses and haven't come across that yet.

I took your suggestion and tried using mb.gof.test on the same model and got this output, which suggests that the model is actually over-dispersed, rather than under-dispersed: 

mb.gof.test(MAWA.elev,nsim=1000,plot.hist = TRUE)

MacKenzie and Bailey goodness-of-fit for single-season occupancy model

Pearson chi-square table:

    Cohort Observed Expected Chi-square

000      0     1797  1795.98       0.00

001      0       22    15.65       2.58

010      0       12    15.65       0.85

011      0       24    22.39       0.12

100      0       14    15.65       0.17

101      0       13    22.39       3.94

110      0       14    22.39       3.14

111      0       64    49.91       3.98

Chi-square statistic = 14.778 

Number of bootstrap samples = 1000

P-value = 0.011

Quantiles of bootstrapped statistics:

   0%   25%   50%   75%  100% 

 0.15  2.88  4.46  6.72 18.99 

Estimate of c-hat = 2.88 

I also calculated the c-hat from the parboot() test that I described in my original post by using cHat_pb <- pb@t0 / mean(p...@t.star), and I got a result of c-hat = 0.72

So this leads me to wonder how these two measures of goodness of fit can result in such opposite values.

Your thoughts are greatly appreciated!

Thanks,

Jenna

[Jenna McDermott]

Hello again John,

I just found a bunch of information that I had somehow missed before and answered my own questions. I'll write them here for anyone who finds this thread. Please correct anything you find to be wrong!

I should not be running parboot using a chi squared statistic on binary data; rather I would need to make some groupings to compare counts instead. Alternatively, the mb.gof.test does exactly this, in its creation of cohorts. So, I should trust the information coming out of the mb.gof.test.

My goodness-of-fit test indicates lack of fit (P < alpha), and the c-hat estimate is not toooo large, so I could use my c-hat estimate to inflate the variances and continue with some conservative inferences. 

Thanks again for leading me to my answers!

Jenna

[John Clare]

Hi Jenna,

It’s difficult to assess dispersion in 0/1 data, so typically this is aggregated to some sort of count: expected vs observed counts of detections across sites or occasions, or the expected vs observed tally of specific detection histories (the MB test). 

I would trust the estimate of overdispersion using the MB test here. If you have volume 1 of the applied hierarchical modeling book, the occupancy chapter talks about this a little more. 

John

On Mar 15, 2021, at 2:23 PM, Jenna McDermott <jmcd...@alumni.uoguelph.ca> wrote:



--
You received this message because you are subscribed to the Google Groups "unmarked" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unmarked+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/2b18fd7e-4a03-46ae-94f2-53f541e18f9fn%40googlegroups.com.

[John Clare]

Yep, sounds about right—beat me to it.

On Mar 15, 2021, at 3:24 PM, 'John Clare' via unmarked <unma...@googlegroups.com> wrote:

 Hi Jenna,

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/658A4D20-5B88-4E23-9D64-58B0A374DD3E%40wisc.edu.