WAIC and LOO giving very different estimates of elpd

[Ben Lambert]

Hi,

I have a problem where LOO and WAIC are giving very different estimates of the expected log pointwise predictive density (elpd). In particular, I get estimates of elpd from LOO-CV that have an absolute value of about an order of magnitude of smaller that that from WAIC; both in terms of point estimates, and the standard errors. I am used to a little difference in these estimates, but not an order of magnitude.

I do get a warning message from the loo package when I estimate LOO-CV on the log-likelihood matrix, that says the following:

Warning messages:

1: 332 (14.1%) Pareto k estimates between 0.5 and 1 

2: 241 (10.3%) Pareto k estimates greater than 1 

I have got these type of messages before, and usually I have ignored them because the latter percentage is small, (and usually the first is small as well.)

My question is, which of these metrics should I trust? I am guessing that I should use WAIC, because it involves less approximation, but I wanted to check.

Best,

Ben

[Andrew Gelman]

Hi, I’m not sure, I’m interested in what Aki says here, but one thing I do notice is your statement “an absolute value of about an order of magnitude of smaller…”.  Unless I’m misunderstanding, this doesn’t make sense.  We look at differences in Waic or Loo, but not absolute values.  That is, Waic=50 compared to Waic=60 is a difference of 10, just like Waic=1050 vs. Waic=1060.  I don’t think you’d take the ratio of these things.  Or maybe you can, but I’m not aware of this ever being done.

A

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Lambert]

Hi Andrew,

Sorry, I was just stating the criteria in absolutes to avoid any issues with interpretation. For the avoidance of further doubt, my results are:

                  Estimate     SE

elpd_loo -15120.1 1270.2

p_loo         8633.3 1215.0

looic         30240.3 2540.4

                   Estimate      SE

elpd_waic -48915.4 12453.8

p_waic       42428.6 12418.2

waic           97830.8 24907.6

Ok, so a factor of 5, on reflection, rather than an order of magnitude. To me, the WAIC results look quite odd - the s.e. is about 25% of the point estimate. The LOO-CV looks more similar to other estimates I have for elpd for other models.

What do you think? I can share the fitted model, although not here, since it has a size of about 2Gb.

Best,

Ben

[Andrew Gelman]

In general, 0 is not a baseline for these things, so the ratio of estimate to se doesn’t really matter.  To put it another way, you could add 100000 to all these estimates, without changing the se’s at all, and the interpretation should not change.

Also, it’s hard to do much with the se for one model; it works better to compute the se for the comparison of 2 models as we do with the arsenic example in our paper.

But are “p_loo” and “p_waic” the effective number of parameters in your model?  Cos those are big numbers for #parameters!  This does make me wonder if you miscoded something!

[Aki Vehtari]

On Wednesday, March 9, 2016 at 12:54:55 AM UTC+2, Ben Lambert wrote:

I do get a warning message from the loo package when I estimate LOO-CV on the log-likelihood matrix, that says the following:

Warning messages:

1: 332 (14.1%) Pareto k estimates between 0.5 and 1 

2: 241 (10.3%) Pareto k estimates greater than 1 

With these values you can't trust PSIS-LOO or WAIC. You could use k-fold-CV.

 

I have got these type of messages before, and usually I have ignored them because the latter percentage is small, (and usually the first is small as well.)

If the latter percentage is larger than zero you should not ignore it.

 

My question is, which of these metrics should I trust? I am guessing that I should use WAIC, because it involves less approximation, but I wanted to check.

WAIC is using a simpler approximation and thus likely to fail before PSIS-LOO.

Sorry, I was just stating the criteria in absolutes to avoid any issues with interpretation. For the avoidance of further doubt, my results are:

                  Estimate     SE

elpd_loo -15120.1 1270.2

p_loo         8633.3 1215.0

looic         30240.3 2540.4

                   Estimate      SE

elpd_waic -48915.4 12453.8

p_waic       42428.6 12418.2

waic           97830.8 24907.6

How many parameters do you have in your model? I guess quite many if the size of the fitted model is 2Gb.

Aki