pcountOpen deciding on a mixture

171 views
Skip to first unread message

Drew Ricketts

unread,
Nov 23, 2015, 10:29:11 PM11/23/15
to unmarked
Hi unmarked Gurus,

When deciding on the best mixture to use for pcountOpen models, is it necessary to perform a goodness of fit test on null models with the 3 mixture options?  Previously, I had been basing my decision about the best mixture for a given set of models on the AIC values for null models with each of the mixture options.  I came across this in the Priol et al. 2014 northern flying squirrel paper.  My models are taking a fair bit of time, and I'd prefer to rely on AIC for mixture selection if that is appropriate.

Thanks a bunch,
Drew

Jeffrey Royle

unread,
Nov 23, 2015, 10:48:27 PM11/23/15
to unma...@googlegroups.com
hi Drew, I think it's ok to use AIC and pick the best model and then I recommend doing a GoF of that model.

The next release of unmarked will have a parboot() function that is parallelized thanks to Adam Smith who wrote some nice code for that. I plan to recompile the whole package this week and test it out but we won't be releasing a new version until Jan or Feb (if you want the pre-release let me know and I'll send it)

regards
andy


--
You received this message because you are subscribed to the Google Groups "unmarked" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unmarked+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andrew Ricketts

unread,
Nov 23, 2015, 10:57:54 PM11/23/15
to unma...@googlegroups.com
Thanks a bunch Andy!  I’d be grateful for the pre-release.

Drew

----------------------------------------
Drew Ricketts
PhD Candidate
Division of Biology
116 Ackert Hall
Kansas State University
Manhattan, KS 66506-4900
aric...@k-state.edu
(785)307-3831
----------------------------------------

You received this message because you are subscribed to a topic in the Google Groups "unmarked" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/unmarked/EK_dfbVTzD8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to unmarked+u...@googlegroups.com.

Evan Cooch

unread,
Nov 26, 2015, 10:17:56 AM11/26/15
to unmarked


On Monday, November 23, 2015 at 10:48:27 PM UTC-5, Jeffrey Royle wrote:
hi Drew, I think it's ok to use AIC and pick the best model and then I recommend doing a GoF of that model.

I disagree. AIC is the sum of the likelihood + number of parameters. Likelihood needs to be corrected for lack of fit for AIC to be meaningful in the first place (moroeoever, where there is structural lack of fit, often number of estimated paramters also needs ot be adjusted). I submit that using AIC to pick the model to test for fit is arguably tautological. 
 


Jeffrey Royle

unread,
Nov 26, 2015, 12:15:32 PM11/26/15
to unma...@googlegroups.com
Evan's point is a good one and worth keeping in mind. On the other hand, it is profoundly unsatisfying from the standpoint of getting something done!

I don't want to put words in Evan's mouth here but one thing people often recommend in place of what I said (testing the best model) is to test the full model (or the saturated model) and then think about model selection if you have a model that fits.   I don't think that is very practical with the Dail-Madsen model because the models are difficult to fit.  But I'm not sure what Evan would recommend... Evan?




Jeffrey Royle

unread,
Nov 26, 2015, 12:28:02 PM11/26/15
to unma...@googlegroups.com
also, happy thanksgiving to everyone!

Andrew Ricketts

unread,
Nov 27, 2015, 12:18:23 AM11/27/15
to unma...@googlegroups.com
Thanks Evan and Andy,

I don’t think I have an issue with the number of parameters, as I am testing 3 constant models to decide which mixture is most appropriate for my data as follows:

(fm1 <-  pcountOpen(~1, ~1, ~1, ~1, pm.umf, K=150))

(fm2 <-  pcountOpen(~1, ~1, ~1, ~1, pm.umf, mixture = "NB", K=150))

(fm3 <-  pcountOpen(~1, ~1, ~1, ~1, pm.umf, mixture = "ZIP", K=150))

I may not have worded my initial question well, and understand that I still need to perform a GoF test on the most parameterized model and use QAIC for model selection if necessary.


Thanks,
Drew

----------------------------------------
Drew Ricketts
PhD Candidate
Division of Biology
116 Ackert Hall
Kansas State University
Manhattan, KS 66506-4900
aric...@k-state.edu
(785)307-3831
----------------------------------------

You received this message because you are subscribed to a topic in the Google Groups "unmarked" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/unmarked/EK_dfbVTzD8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to unmarked+u...@googlegroups.com.

Evan Cooch

unread,
Nov 30, 2015, 8:11:53 AM11/30/15
to unmarked
Been away from email for a bit (family holiday and all that). 


On Thursday, November 26, 2015 at 12:15:32 PM UTC-5, Jeffrey Royle wrote:
Evan's point is a good one and worth keeping in mind. On the other hand, it is profoundly unsatisfying from the standpoint of getting something done!

I suspect this won;'t come across well, but...better to have it right, than fast. Expediency is an operational consideration, but since the premise is we're doing 'science', then inconveniences on doing it right trump the 'gains' in getting 'a result' quickly. Personally, I;'d rather have the r9ight result, than the 'fast result'. In a perfect worlkd, they're the same. 

 

I don't want to put words in Evan's mouth here but one thing people often recommend in place of what I said (testing the best model) is to test the full model (or the saturated model) and then think about model selection if you have a model that fits.   I don't think that is very practical with the Dail-Madsen model because the models are difficult to fit.  But I'm not sure what Evan would recommend... Evan?


This is, of course, a huge issue at the moment. Our capacity to 'construct' a model, and 'derive estimates' from it has in many instance exceeded our capqability to assess fit of the model to the data in the first place. Sometimes, the problem is conceptual -- a saturated model isn't always defined. In such cases, unclear how to proceed. In other cases, there are numerically intensive resampling approaches which may be the best you can do (the median c-hat,in MARK, derived from some early suggestions by eorge Sevber, is an example of same). The problems with GOF testing are many -- is the lack of fit due to structural incomaptability of the model with the 'true' structure of the data, or is it because of poorly beaving 'animals' who don't understand that they should all flip completely independent multinomial coins for various life events. In my experience, there is a subtle but important conflict between the two -- in the latter case, most folks seem content with estimate some sort of measure of lack of fit (the ubiquitous c-hat), used in some sort of quasi-likelihood adjustment to model selection (via AIC or otherwise). The estimation of c-hat becomes an estimation problem. But, the estimation is not informative -- it doesn't give any clues as to 'why' there is a lack of fit, merely, it estimates the magnitude of the lack of fit. Without knowing why there might be lack of fit, its hard to know if the model structure, given the data, is flawed from the outset. In other words, there is information in lack of fit that might be important (i.e., it isn't like all due to extra-binomial noise). 

So, what to do? In the absence of a reasonable abiulity to estimate/parse lack of fit (note: because it might take a long time on a computer does *not* constitute reasonable -- compute time is cheap, realtive to the cost of collecting the data. If it takes hours -> days to run a model, so what, other than the risk that your system crashes before its done), what I've often suggested, which is admitedly ad hoc, but which at least has some conceptual basis, is to evaluate the 'sensitivity' of the model selection criterion (say, randkings based on AIC) to different values of c-hat. Default is 1. Then, you try 1.1, 1.2...2.0 or so. If the basic rank order of models stays more or less constant, you're probably fine/safe. If, on the other hand, you get 'big' jumps in rank order, especially for c-hat in the 1.0 -> 2.0 range, then I'd be pretty suspicious. I did a fairly large simulation experiement at one point that showed that big jumps in rankings often strongly indicated basic structural causes for lack of fit. 

As a (somewhat) related aside -- as c-hat goes up, various 'rules of thumb' for using relative normalized AIC weights as the basis for model selection require more thought (since everything gets 'more conservative'). This is treated in some detail in Edwards 'likelihood' book, and in precis form in the MARK book in a couple of places. 

At any rate...

Evan Cooch

unread,
Nov 30, 2015, 10:22:28 AM11/30/15
to unmarked



As a (somewhat) related aside -- as c-hat goes up, various 'rules of thumb' for using relative normalized AIC weights as the basis for model selection require more thought (since everything gets 'more conservative'). This is treated in some detail in Edwards 'likelihood' book, and in precis form in the MARK book in a couple of places. 


Whoops - I meant Richard Royall's book on 'Statistical Evidence' (not the Edwards book). 
 
 
Reply all
Reply to author
Forward
0 new messages