Poisson vs Negative Binomial aster predictions

[John Benning]

Hi all,

I'll start with the caveat that the answer to my question may come down to "please read the technical reports" or "learn more maths," so apologies if I'm being dense!

I'm puzzling over some (what seems to me at least) strange behavior when dealing with a final graph node being modeled as Poisson vs. negative binomial. These data are plant lifetime fitness data from a common garden experiment. For this simplified example, we're simply interested in understanding the effect of source population Latitude (a continuous predictor) on lifetime fitness (here, the number of fruits produced by the plant). 

I've attached data, R code, and an HTML analysis output file.  

The aster graph has three nodes (plus root): 1 --> Survival at 4 weeks --> Any Fruits --> Number of Fruits. The first two nodes are Bernoulli. As you can see in the attached files, the final node probably "should" be modeled as zero-truncated negative binomial. However, we will eventually need to incorporate a random effect into these aster models (to deal with the pseudoreplication within source populations), and reaster, as I understand it, cannot be used with negative binomial nodes. So I also tried modeling the final node as zero-truncated Poisson. I thought this may change the estimates and SE's a bit, but perhaps was "worth it" in order to use reaster. (Just FYI, running just ordinary Poisson and NegBinom regressions on the subset of plants that made fruits produced fairly similar estimates, though with quite different SEs.)

However, using Poisson completely changed the relationship between Latitude and fitness in, what seemed to me, a very strange way. Further details in the R code, but to quickly illustrate: 

Here are the raw data:

Here are the predictions from an unconditional aster model with the final node as ZT NegBinom:

and here are the predictions from an unconditional aster model with the final node as ZT Poisson:

This second plot seemed strange to me. In trying to figure out what was going on, I tried running the ZT Poisson model as a conditional aster model, which resulted in this:

which obviously seems more "correct."

So my question is, does anyone know what's going on here? Why does the conditional model with final node as ZT Poisson approximate the unconditional model with final node as ZT NegBinom so much more closely than the unconditional/ZT Poisson model? What would be the major drawbacks of using the conditional model instead of the (oft-recommended) unconditional model?

Thanks for any insight!

John

[John Benning]

Google isn't letting me upload attachments for some reason...so they are in this folder.

Sorry!

John

[geyer]

I have carefully looked this over and do not see any mistakes in your coding of fitting and plotting for zero-truncated negative binomial and zero-truncated Poisson.  Assuming there are no mistakes that I missed, I would say this is just what maximum likelihood does for these models.  You know the zero-truncated Poisson model does not come close to fitting the data, so it should be no surprise when its fits are radically different.  It sharp rise in the mean fitness curve for that model is no surprise.  That is where the Bernoulli probabilities are rising.  The only surprise is that you do not see that on the zero-truncated negative binomial plot (is is off the right edge).  Why the Poisson part of the curve is so flat I do not understand but I see no reason to attribute it to bug in code.   An experiment that I did not try is to try different alpha for the negative binomial.  As alpha to infinity, negative binomial converges to Poisson and the curves in the plot at the end of the attachment show that your two curves have a continuum of curves in between going smoothly from one to the other.  So this is indeed just what maximum likelihood estimation in aster models does.

As to your other issue about negative binomial (truncated or not truncated) being incompatible with random effects, that is correct.  This has nothing to do with aster models.  That is just a problem with the negative binomial distribution.  Its parameter space is minus infinity to zero (not the whole real line).  Hence it cannot support normal random effects.

Presumably, the right thing to do to get random effects and overdispersed Poisson is to have individual effects for the overdispersion plus any other random effects you want.  But then the Laplace approximation (Breslow-Clayton) scheme used in R function reaster does not work (it will be a very bad approximation to maximum likelihood).  So some MCMC scheme would have to be used.  We have tried such but have not published anything yet.  We have not managed to make it safe for ordinary users to use.

So at this point you are stuck on that AFAICS.

[John Benning]

Thanks so much for the quick response, Charlie! This is very helpful, especially seeing your last plot. I shouldn't have assumed I'd get an answer that looks "right(ish)" when Poisson fits the distribution so poorly.

Cheers,

John

[Charles Geyer]

The theory of misspecified maximum likelihood (http://www.stat.umn.edu/geyer/8112/notes/equations.pdf) says it does as well as it can do with the model it is given.

--
You received this message because you are subscribed to the Google Groups "Aster Analysis User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aster-analysis-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aster-analysis-user-group/854c7681-47cf-4348-aaf6-7813de3839e1n%40googlegroups.com.

--

Charles Geyer
Professor, School of Statistics
Resident Fellow, Minnesota Center for Philosophy of Science
University of Minnesota
cha...@stat.umn.edu

[geyer]

Further thoughts on whether negative binomial and zero-truncated negative binomial can be used with R function reaster (aster models with random effects).  The warning given by R function reaster about negative binomial is just a warning.  It can be ignored.  Here is an argument for ignoring it.  The algorithm used, somewhat similar to Breslow-Clayton, does not actually calculate the correct log likelihood.  What it does will not crash if negative binomial or zero-truncated negative binomial families are in the model.  So just ignoring the warning is undeniably a TTD (thing to do).   OTOH, here is the argument against ignoring it.  There is no statistical model backing up the TTD (because of exactly what the section about this in help(reaster) says).  So it is somewhat unclear what this TTD would be doing other than it does whatever it does.  So I am not completely convinced that this TTD is hopeless.  Presumably that is why I made it a warning rather than an error.  But it is very unclear what validity, if any, this TTD has.