Using ANOVA with ELPDs to choose between models

[Ben Lambert]

Hello all,

I have been following the method of Vehtari et al. (2015) to compare models; specifically eqn. 24.

At the moment I am using this criterion by computing (for models A and B) a statistic:

                L =   (total(ELPD_A) - total(ELPD_B))/se(ELPD_A-ELPD_B)

If this number is much greater than 1, then we prefer model A.

First, is this method correct? For example, consider the below estimated ELPDs:

ELPD_A = (-100,-200,-200,...,-300,-400)

ELPD_B = (-100,-201,-201,...,-300,-400)

where "..." is the same for these two models, and let's say, of length 95 for both models. In this case, I think (again correct me if I'm wrong) that the se(ELPD_A-ELPD_B) =sqrt(100 x var(ELPD_A-ELPD_B)) ~ 1.4.

The difference in total ELPD ~ 2 > 1.4. Therefore we prefer model A. Is this logic correct?

So, if the above is right, this method appears to sometimes mean that we 'prefer' one model over another because a small number of estimated ELPD scores are very slightly higher. 

Second, if we were to use ANOVA to compare the means between groups A and B, would this be ok? I have tried this and it seems to give a more reasonable answer that A and B are indistinguishable. Moreover, could this method be used across a number (greater than 2) of models?

Perhaps I am getting mixed up on two different viewpoints for comparing A and B: the first is purely in terms of point by point prediction, in which case A wins; the second is asking if there is any practical difference in the fits. Forgive the ignorance!

Best,

Ben

[Aki Vehtari]

On Wednesday, April 13, 2016 at 4:58:03 PM UTC+3, Ben Lambert wrote:

Hello all,

I have been following the method of Vehtari et al. (2015) to compare models; specifically eqn. 24.

At the moment I am using this criterion by computing (for models A and B) a statistic:

                L =   (total(ELPD_A) - total(ELPD_B))/se(ELPD_A-ELPD_B)

If this number is much greater than 1, then we prefer model A.

First, is this method correct? For example, consider the below estimated ELPDs:

ELPD_A = (-100,-200,-200,...,-300,-400)

ELPD_B = (-100,-201,-201,...,-300,-400)

where "..." is the same for these two models, and let's say, of length 95 for both models. In this case, I think (again correct me if I'm wrong) that the se(ELPD_A-ELPD_B) =sqrt(100 x var(ELPD_A-ELPD_B)) ~ 1.4.

The difference in total ELPD ~ 2 > 1.4. Therefore we prefer model A. Is this logic correct?

With that difference P(A is better than B)~0.92. There are different views whether this probability is high enough to say "prefer".

So, if the above is right, this method appears to sometimes mean that we 'prefer' one model over another because a small number of estimated ELPD scores are very slightly higher. 

But yes, in this approach it is possible that small difference is enough to prefer one model if that model is consistently better (in this case equal or better for all loo cases).

It is an approach for answering the question which has better performance. It is good to think what is your prediction task and what utility (or cost) function you want to use. You can also look at the magnitude of the difference, but it is not so easy to decide what would be practical difference in log density scale. It might be helpful to then use some application specific utility (or cost) function where "practical difference" is calibrated by the expert knowledge. We are writing a paper illustrating some options.

 

Second, if we were to use ANOVA to compare the means between groups A and B, would this be ok? I have tried this and it seems to give a more reasonable answer that A and B are indistinguishable. Moreover, could this method be used across a number (greater than 2) of models?

If you follow the logic that uncertainty is related to not knowing the distribution of the future data and we approximate it by the observed data, then the comparison has to be a paired comparison.

Aki

[Ben Lambert]

Thanks Aki. Much appreciated. I agree that it would be good to have some guidance regarding the tangible benefits of one model versus another grounded in decision theory. Look forward to reading the paper!

Best,

Ben