K-fold cross-validation/Future predictions for hierarchical models

[Olive Buhule]

Hello Stan Users,

This may be a stupid question but I am a bit confused. In K-fold cross-validation and/or future predictions, how do you deal with random effects in hierarchical models given that, they are individual specific? In short, if I have a held-out set do I have to estimate the random effects for this set or use the ones estimated in the training set? And if I have to, how do I go about it?

For instance, I have a model with 4 random effects estimated as:

b~multi_normal_cholesky(mu_b, L_Sigma_b); with priors on mu_b and L_Sigma_b.  Should I do the same for the held-out set?

Thank you for your kind response.

 

Olive

[Michael Betancourt]

This gets to a subtlety in predictive comparisons in hierarchical models.  There are

two interpretations of prediction — either you can sample new data conditioned on the

same groups, or you can sample new groups from the hierarchical population distribution

and then new data from those groups.  Ultimately this is related to whether the random

effects are considered part of the prior (in which case predictions use set the random

effects to their inferred values) or part of the likelihood as missing data (in which case

predictions sample new random effects and then data conditioned on those).  

If your random effects are individual specific then you’ll have to sample new random

effects from the hierarchical population distribution and then new data from those

random effects.  This will give you a predictive distribution that you can compare to

any new data.

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Gelman]

Yah, we discuss this in chapter 6 and in our 1996 paper.

[Olive Buhule]

Thank you so much for the clarification.

Olive