Question regarding K-fold on hierarchical model

[Aki Vehtari]

Moved from github psis repo issue https://github.com/avehtari/PSIS/issues/4 to here

I'm trying to use K-fold crossvalidation for a hierarchical model where each subject has a subject-level parameter (theta[i]), that comes from a group-level parameter (theta_group ~ Normal).
Each subject has 100+ observations, and the likelihood for each subject is the sum of likelihood at each observation given the parameter theta[i] of the subject.
I am running 10-fold crossvalidation by excluding 10% subjects from the training data set in each fold.
However when trying to estimate the fit of the model in the testing dataset (that is, on the 10% of subjects that were excluded from the training dataset -- cvitst), what parameters should I use to calculate the likelihood of each subject in this testing dataset ? I can see in the 'demo_stan_arsenic' that we reuse the model parameters sampled in the testing dataset to estimate the likelihood at these points in the testing dataset. However, I'm not sure how that would apply with multiple individual subjects that each have different parameters.
So far I've thought of using the mean parameter of all subjects as the parameter to 'plug-in' in the generated quantities for the likelihood of the testing dataset (cvitst) but not sure this is the right way to do it :-/
Any pointers would be super appreciated :+1:
Best,
Feezdev

If you have a hierarchical model with user specific parameters, you make predictions for a new user by using parameters sampled from the corresponding prior. You can do this easily in generated quantities block.

Aki

[Feezdev]

Thank you so much Aki for the reply (and terribly sorry for posting in the wrong place -- my bad).

Just to make sure I'm getting this correctly (I hope the question isn't too silly), given a model of the sort:

model {

    alpha_l ~ normal();    //hyperprior shape param
    beta_l ~ normal();      //hyperprior shape param


    theta_t ~ normal();     //hyperprior scale param
    kappa_t ~ normal();   //hyperprior shape param


    lambda ~ beta(alpha_l,beta_l)   //prior lambda param
    tau ~ gamma(kappa_t,theta_t)  //prior tau param


    for (i in 1:N) {                             //per subj
           for (ii in 1:(nt[i])) {               //per trial
                  y[i,ii] ~ categorical_logit( do stuff with tau[i], lambda[i] and x[i,ii] );
           }
    }
}

Would you suggest I do K-fold using '_rng' s in the generated quantities to generate parameters at random given the hyper parameters (as in below):

generated quantities {
         real cv_ll[N];
         
         real tau_cv[N];
         real lambda_cv[N];
       

         for (i in 1:N_cv) {


                      cv_ll = 0;
                      lambda_cv = beta_rng(alpha_l,beta_l);         // pick random lambda value from prior given hyperprior values at current 'generated quantities' sample
                      tau_cv = gamma_rng(kappa_t,theta_t);        // pick random tau value from prior given hyperprior values at current 'generated quantities' sample


                      for (ii in 1:T) {
                             cv_ll[i] = cv_ll[i] + categorical_logit_lpmf( do stuff with tau_cv[i], lambda_cv[i] and x_cv[i,ii]  );
                      }
         }
}

Or take the mean of the prior given the current sample of the hyperpriors? (see below)

generated quantities {
         real cv_ll[N];
         
         real tau_cv[N];
         real lambda_cv[N];
       

         for (i in 1:N_cv) {


                      cv_ll = 0;
                      lambda_cv = (alpha_l / (alpha_l + beta_l));   // Take E(lambda | alpha_l, beta_l) given current hyperprior parameters at 'generated quantities' sample
                      tau_cv = (kappa_t * theta_t);                         // Take E(tau | kappa_t, theta_t) given current hyperprior parameters at 'generated quantities' sample


                      for (ii in 1:T) {
                             cv_ll[i] = cv_ll[i] + categorical_logit_lpmf( do stuff with tau_cv[i], lambda_cv[i] and x_cv[i,ii]  );
                      }
         }
}

Thank you again so much for your help,
Best,

[Bob Carpenter]

We tend to do the first thing---calculate the predictive posterior
distribution and measure that. That means if there are item-level
latent parameters, they need to be randomly generated and then used
for prediction.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Aki Vehtari]

On Tuesday, January 24, 2017 at 11:57:34 PM UTC+2, Bob Carpenter wrote:

We tend to do the first thing---calculate the predictive posterior
distribution and measure that.  That means if there are item-level
latent parameters, they need to be randomly generated and then used
for prediction.

Yes.

Aki

[Feezdev]

Aki, Bob,

Thank you so much for your answers this is so helpful ! I'll do just what you recommend doing :)

Again thanks a million times!