PSIS-LOO for nested/hierarchical cross validation

[Luigi Acerbi]

I am doing nested (or hierarchical?) cross-validation.

Let's say that I have two model classes M1 and M2, and in each class there are several models with different number of discrete components k (e.g., components in a mixture model).

I want to infer what's the best model class for a given dataset.

I could just have two super-models and do full MCMC directly in the larger space (treating the number of components as discrete parameters), but that's not viable for various reasons.

So, I am computing the posterior, via MCMC, separately for each model class M and number of components k.

The question is, is it okay to use PSIS-LOO twice here?

As far as I understand, I should do something like two nested leave-one-out CV:

1. For each model class M,
• For i = 1..N, where N is the number of data points:
  
• remove the i-th data point from the full dataset.
• Perform PSIS-LOO (or any other method) on the reduced dataset to choose the best number of components.
• Take the best model and now compute the prediction on the left-out i-th data point via PSIS. This yields the i-th LOO-CV score for the model class.
• The LOO-CV score of each model class is the sum (or mean) of 1..N LOO-CV scores from the N reduced datasets.
2. Compare the summed scores of the model classes.

Does this make sense, and is there a reference for an usage of this kind? Any non-obvious caveat?

(I was checking this but it does not seem to cover this case.)

Thanks,

Luigi

[Aki Vehtari]

Hi Luigi,

Yes, it is sensible to use nested cross-validation here (although

integration over k would be better).

You can't use nested PSIS, because in the inner loop you are

doing model selection (optimisation of k). Importance sampling in PSIS

can be used only when you have samples from the posterior.

I recommend using K-fold-CV (e.g. K=10) in the outer cross-validation.

Not also that, the inner optimisation of k will cause overfitting and

the outer cross-validation is only estimating the amount of

overfitting, but can't prevent the overfitting. In some cases the

overfitting can be really bad. See Piironen & Vehtari, Comparison of

Bayesian predictive methods for model selection, which has been just

published in Statistics and Computing

http://link.springer.com/article/10.1007/s11222-016-9649-y

Aki

[Luigi Acerbi]

Hi Aki,

thank you for your reply.

You can't use nested PSIS, because in the inner loop you are

doing model selection (optimisation of k). Importance sampling in PSIS

can be used only when you have samples from the posterior.

So, I can definitely use PSIS-LOO for model selection (choice of k) in the inner loop, since I have the posterior for each k.

I understand that I also need a full posterior for PSIS-LOO in the outer loop, but I guess I could still use an approximate posterior?

Possible approximations are:

1. I just keep the model I select (i.e., the posterior over k has collapsed onto a single model). This is probably bad...
2. As a possibly better alternative, I can build a rough estimate of the posterior over k based on the exponentiated PSIS-LOO CV score of each k (similar to what Burnham & Anderson suggest for AIC-weighted prediction). Then I can easily produce samples from this "pseudo-posterior". I understand that this is not correct -- it's not even a proper approximation of the marginal likelihood (the log is on the wrong side of the summation, and I am using the posterior over parameters as opposed to the prior). However, it looks like that this pseudo-posterior would have some desirable properties in common with the marginal likelihood (i.e., it punishes bad models, and applies some sort of Occam's razor to complex models).

Supposing that (2) is not insane, could I then proceed with PSIS?

I recommend using K-fold-CV (e.g. K=10) in the outer cross-validation.

Yes, that's the alternative (and long) way to go that I had in mind.

 

Not also that, the inner optimisation of k will cause overfitting and

the outer cross-validation is only estimating the amount of

overfitting, but can't prevent the overfitting. In some cases the

overfitting can be really bad. See Piironen & Vehtari, Comparison of

Bayesian predictive methods for model selection, which has been just

published in Statistics and Computing

http://link.springer.com/article/10.1007/s11222-016-9649-y

Oh, this is very serendipitous, thanks!

I will start reading right now.

Thanks,

Luigi 

[Luigi Acerbi]

1. I just keep the model I select (i.e., the posterior over k has collapsed onto a single model). This is probably bad...
2. As a possibly better alternative, I can build a rough estimate of the posterior over k based on the exponentiated PSIS-LOO CV score of each k (similar to what Burnham & Anderson suggest for AIC-weighted prediction). 

PS: I see why (1) would not work. If a LOO posterior ends up selecting a different model than the full posterior, then there is no overlap between posteriors, and clearly IS would not work.

However, as bad as it can be, (2) would not have this problem. Perhaps I wouldn't even need to actually sample k, I could use the formula of the pseudo-posterior when needed, although I haven't thought about the maths in detail.

[Aki Vehtari]

On Monday, April 11, 2016 at 12:02:58 AM UTC+3, Luigi Acerbi wrote:

1. As a possibly better alternative, I can build a rough estimate of the posterior over k based on the exponentiated PSIS-LOO CV score of each k (similar to what Burnham & Anderson suggest for AIC-weighted prediction). Then I can easily produce samples from this "pseudo-posterior". I understand that this is not correct -- it's not even a proper approximation of the marginal likelihood (the log is on the wrong side of the summation, and I am using the posterior over parameters as opposed to the prior). However, it looks like that this pseudo-posterior would have some desirable properties in common with the marginal likelihood (i.e., it punishes bad models, and applies some sort of Occam's razor to complex models).

It might work or not. I don't know.

LOO (and similar approaches) doesn't penalize complexity if the predictions are good.

 

Supposing that (2) is not insane, could I then proceed with PSIS?

Maybe. I don't know.

 

I recommend using K-fold-CV (e.g. K=10) in the outer cross-validation.

Yes, that's the alternative (and long) way to go that I had in mind.

K-fold-CV is probably much faster than showing that pseudo-thing works reliably. You would probably need to compare pseudo-thing to K-fold-CV anyway.

Aki