issues calculating WAIC

[Kirsty Laurenson]

Hi all, 

I am using nimble to run multistate mark-recapture models for seabirds, as I wish to estimate survival and recruitment of juvenile, immature, and adult birds, and eventually introduce environmental covariates to investigate drivers of survival and recruitment. However,  I am having issues calculating WAIC when I include 'WAIC = TRUE' in the 'nimbleMCMC' call. I get the same WAIC value of 0.000001579057 for two different models (phi, psi and p constant, and phi time-dependent, but psi and p constant). Other than this, the models give me reasonable estimates for the parameters, and appear to have converged and chains mix well. I don't understand why the WAIC isn't being calculated, as I can calculate WAIC for CJS models using single-state data.

My code is attached, and I would be very grateful for any information or advice as I am a bit of a nimble novice!

Thanks in advance, 

Kirsty

[Chris Paciorek]

Hmm, that is indeed odd. WAIC is based on the log density of the data (`y` in your case) for each posterior sample, so if the density values differ between 

the different runs, which they presumably do, then WAIC should as well.

If you could share the data file, or an Rda file with `y`, I will take a deeper look. You're welcome to share it off-list with me if you prefer (or provide a facsimile dataset in the same format if the data are confidential).

-chris

-chris

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/6834fccb-b20a-4918-8561-834622d8d3f6n%40googlegroups.com.

[Chris Paciorek]

Hi Kirsty (following on some off-list discussion),

The problem here is that the likelihood is not changing as the parameters change. The reason is that the way the model 

is defined, `z[i,t]` says which row of `omega` is used for `y[i,t]`. As the MCMC proceeds, a given `z[i,t]` can only 

be such that the observed `y[i,t]` is possible, and the probability vector for the `dcat` for `y` has a probability of one 

in one position and zero in the others. So the probability density for y is always 1 (log probability density always 0) for 

every MCMC iteration (i.e., regardless of the current parameter values) since values of z that don't make this be the case

would be rejected.

I started to think about the marginal version of WAIC, averaging over possible `z` values, but if one simulates `z` values

based on current parameter values in the MCMC but not conditioning on the data (as would happen with marginal WAIC), 

one will produce `z` values that cause the likelihood to be 0. So I don't think that will work. 

So ultimately, I think there's a statistical question in terms of how to apply  WAIC to this model structure, not a nimble issue.

Also, this is perhaps a side note but perhaps relevant --  I'm not understanding how the model can give a valid (non-zero) likelihood for the y's that 

are equal to 2. None of the rows of `omega` have positive probability on the second element. I must not be seeing/understanding

something. 

As you may have noticed, the value of `pWAIC` that nimble is returning is 0 and the value of `lppd` is essentially zero 

(seemingly up to some floating point numerical imprecision). This reflects the issue above and is the reason you get the same

"WAIC" for both models. 

-chris