comparing models using waic

[Linas Mockus]

Hi,

I think it is a very simple question but how to compare two different models for the same data using waic when response for one model is scaled (a drastically simplified example):

    for (i in 1 : N) {
         log_lik [i] <- lognormal_log (response [i], input [i], sigma);
    }

and

    for (i in 1 : N) {
        response_scaled [i] <- response [i] / input [i];
         log_lik [i] <- lognormal_log (response_scaled [i], input [i], sigma);
    }

Thanks for any advice,
Linas

[Andrew Gelman]

Hi—As long as the data haven’t changed, you can compute the loglikelihood of the original data points and it should all work.  I think the key is that ultimately you are modeling the data themselves, not the transformed or combined data.

A

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastian Weber]

Hi!

Not sure if I understand Andrew. Assume we have data y and some model for it. Let´s say we have two non-linear transforms f and g which we believe does some good to the data in terms of whatever (better variance structure or so) - but I am not sure if I want to model f(y) or g(y). Hence, in the Stan model I then write either

log_lik [i] <- normal_log (g(y[i]), pred[i], sigma);

or

log_lik [i] <- normal_log (f(y[i]), pred[i], sigma);

to compare the model with f and g respectively (each model is executed independently from the other one).

To my understanding this is how it has to be done, but it feels certainly a bit odd to compare the point-wise log-density based on different data effectively. However, this is the best one can do in this case as such a model lacks a link function. Right? I am asking because in such a scenario, one does not calculate the loglikelihood on the original data, but rather the transformed one - this is the part which confused me in your response.

Best,

Sebastian

[Andrew Gelman]

Hi, Sebastian. My point is that in both cases the model is (or could) ultimately be expressed as

y[i] ~ …

Or, to put it another way, there is a term in the log-likelihood of the form p(y[i]|theta).

[Avraham Adler]

Sebastian, if I understand you correctly, you are modifying the raw data to get a better fit. However, once the model is run, the parameters can be restated to be on the "data scale," and as the data is the same, the WAICs calculated by using the different sets of parameters rescaled back to the data ON the same data are now comparable.

Avi

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

[Sebastian Weber]

Hi!

Avi, yes I transform my raw data with either f or g and then run the model on the transformed data. So if I get you right, then I have to work a little more in order to do a WAIC comparison of the models as I need to compare based on the original data and hence need to get from p_f(y) the p_y (confusing notation here sorry). Ok, makes sense as I do care about the fit on the original data scale rather than the tricky transformed data.

Many thanks for clarifying.

Best,

Sebastian