Quantifying the predictive capability of two models in a way consistent with ELPD

[Ben Lambert]

Hello all,

I wrote a previous message regarding when model A has a better predictive capability than model B (using the Vehtari 2015 paper). I think I now understand why we need to use the paired comparison in ELPD between the two models.

However, I am also looking for a measure to quantify (in plain English) the predictive capability of the two models. Basically something along the lines of "model A explains X% of the variance, and model B explains Y% of the variance".

The problem is that, when I use simple R-squareds to compare the two models, I frequently get a contradiction with the ELPD results. In other words:

by ELPD: model A > model B

by R-squared: model A < model B

where '>' and '<' indicate 'better than' or 'worse than' respectively.

This isn't surprising to me, since they measure completely different things. However, it is quite unfortunate since I want to quantify (ideally via a percentage measure, or something that can be interpreted biologically), the improvement in model A relative to model B. 

Does anyone have any ideas for what I could use that would give a consistent story? (Another measure would be something like % deviance explained, but am unsure how to estimate this.)

Best,

Ben

[Bob Carpenter]

This isn't a problem that can be answered abstractly.

What are the quantities of interest for the model? I'd use
that angle to talk to consumers of model predictions. It
also gives you a focus for predictive posterior comparisons.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Lambert]

Ok, apologies for the lack of detail in the previous message. I have a hierarchical model of within-host malaria (their viral load), and the different hierarchies correspond to different treatment groups.

I am using various host covariates to explain the viral load within a patient. I have a number of models for this, each representing different biological assumptions about the disease. I want the data to allow me to resolve a difference between these models.

As such, I have explicitly done k-fold cross validation to estimate the out-of-sample predictive performance of each model. This is the ELPD to which I was referring in my original question.

However, I would also like to state something in biological/numerical terms which allows a person to get a sense of just how much variation is being explained by each model. This is what I was referring to in my original question when I said, model A explains X% of the variance, and model B explains Y% of the variance (in viral load)".

The problem I have at the moment is that the two different metrics - ELPD vs % variance explained - are giving conflicting views. Part of me thinks that - if all I ultimately care about is % variance explained - then I should use this to compare the models (and forget about ELPD).

Does this sound reasonable, and the best approach?

Best,

Ben

[Jonah Gabry]

Hi Ben,

In general any model B that has more parameters than model A will "explain" (in a not very useful sense of the word) more of the variation in the outcome. It's unfortunate that "explain" is the traditional term used here because model B might not explain much of anything useful and will still have a higher R^2. Another way to say this is that comparing models based on proportion of variance "explained" is asking the question "which model explains more?", but you're interested in a different question, that is "which model explains better?".

ELPD has its limitations, but it comes closer to answering the quevstion you will typically want. As you say, the cost of this is that it's less intuitive and less direct than "explained" variation because computed ELPDs only have meaning relative to other computed ELPDs.

Maybe there's a clever way to present it for your particular application (which sounds interesting by the way) but I still the a better method with more difficult presentation is preferable to inferior method with simple presentation.

Jonah

[Aki Vehtari]

Great answer by Jonah about R^2. 

Ben, you write that you want to estimate "the viral load within a patient". Is it important that this prediction has small absolute error or small relative error or something else?

Aki

[Seth Flaxman]

Since you're doing k-fold crossvalidation, can't you calculate something like % variance explained on the held out data, which would hopefully be both interpretable and consistent with EPLD? 

--

[Ben Lambert]

Hello all, 

Thanks for all your comments.

Jonah - I'm not sure I follow your R^2 comment; I am calculating R^2 on the hold-out dataset (sorry if this wasn't clear). As such, the number of parameters shouldn't matter, only the ability of the model to generalise. In other words, there shouldn't be the type of overfitting issues that plague R^2 usually.

Seth - this is what I am already doing. The trouble I am having is that the % variance explained is not agreeing with the ELPD estimates.

Aki - it's really the relative fit that I care about. This was the reason for wanting to use % variance explained by the model.

Again, I want to emphasise that I am doing all these calculations on hold-out datasets. 

Best,

Ben

[Ben Lambert]

Hello all,

An idea occurs to me here: I could calculate the log density of the actual data given parameters predicting that data exactly (a sort of 'saturated' model). This would give a sort of 'best case' to which I could compare log(y|theta ~ posterior). This is similar to the % deviance explained that is used to evaluate ML models.

Does this sound reasonable? Anyone else got a better idea?

Best,

Ben

[Jonah Sol Gabry]

On Wednesday, April 20, 2016, 'Ben Lambert' via Stan users mailing list <stan-...@googlegroups.com> wrote:

Hello all, 

Thanks for all your comments.

Jonah - I'm not sure I follow your R^2 comment; I am calculating R^2 on the hold-out dataset (sorry if this wasn't clear). 

Yeah when I wrote that I was thinking about the interpretation of R^2 in general.

[Aki Vehtari]

In the original DIC paper by Spiegelhalter et al they did use this saturated model approach. But since it doesn't change the comparison, I haven't seen it much used after that.

Aki